2024-09-10T18:45:19.175 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T18:45:19.191 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T18:45:19.280 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900276 branch: wip-guits-quincy-2024-09-09-1810-quincy description: orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} email: gabrioux@ibm.com first_in_suite: false job_id: '7900276' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-quincy-2024-09-09-1810-quincy ceph: conf: client: debug ms: 1 global: mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: cephadm-package install: ceph: flavor: default sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 extra_packages: - cephadm selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: quincy-orch-centos-9-teuth sha1: fad5356a94e6044ce1062d748daa5a0f2ed53a7d owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 4625 sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: quincy-orch-centos-9-teuth suite_path: /home/teuthworker/src/github.com_adk3798_ceph_fad5356a94e6044ce1062d748daa5a0f2ed53a7d/qa suite_relpath: qa suite_repo: https://github.com/adk3798/ceph.git suite_sha1: fad5356a94e6044ce1062d748daa5a0f2ed53a7d targets: smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMpqcrPM02VsUQNDISD/urTVbclUv3+PJkoKoinPvXdUF/62QaIosSbJPeP/1frp3o80RDt0R91U0HO2zUMAyXw= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPpH1qfsYcWz9FlfeBMpcIQXNA2r3d/0srGivpKlD1uZhEbuH6qJnfElWu0gAjE5UiDChFqJH7JsbFwMm/uRBc8= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_17:53:37 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-10T18:45:19.280 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_adk3798_ceph_fad5356a94e6044ce1062d748daa5a0f2ed53a7d/qa; will attempt to use it 2024-09-10T18:45:19.281 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_adk3798_ceph_fad5356a94e6044ce1062d748daa5a0f2ed53a7d/qa/tasks 2024-09-10T18:45:19.281 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T18:45:19.283 INFO:teuthology.task.internal:Checking packages... 2024-09-10T18:45:19.304 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5be57395ea235535910512ae73ba8e7ff25fe588' 2024-09-10T18:45:19.304 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T18:45:19.304 INFO:teuthology.packaging:ref: None 2024-09-10T18:45:19.305 INFO:teuthology.packaging:tag: None 2024-09-10T18:45:19.305 INFO:teuthology.packaging:branch: wip-guits-quincy-2024-09-09-1810-quincy 2024-09-10T18:45:19.305 INFO:teuthology.packaging:sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:45:19.305 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-quincy-2024-09-09-1810-quincy 2024-09-10T18:45:19.447 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1606.g5be57395 2024-09-10T18:45:19.448 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T18:45:19.455 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T18:45:19.455 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T18:45:19.469 INFO:teuthology.task.internal:Saving configuration 2024-09-10T18:45:19.480 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T18:45:19.488 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T18:45:19.512 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900276', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 18:40:27.324181', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMpqcrPM02VsUQNDISD/urTVbclUv3+PJkoKoinPvXdUF/62QaIosSbJPeP/1frp3o80RDt0R91U0HO2zUMAyXw='} 2024-09-10T18:45:19.537 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900276', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 18:40:27.325464', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPpH1qfsYcWz9FlfeBMpcIQXNA2r3d/0srGivpKlD1uZhEbuH6qJnfElWu0gAjE5UiDChFqJH7JsbFwMm/uRBc8='} 2024-09-10T18:45:19.537 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T18:45:19.544 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-10T18:45:19.544 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-10T18:45:19.544 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T18:45:19.595 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbe36afbac0>, signals=[15]) 2024-09-10T18:45:19.595 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T18:45:19.602 INFO:teuthology.task.internal:Opening connections... 2024-09-10T18:45:19.602 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-09-10T18:45:19.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:45:19.678 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-09-10T18:45:19.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:45:19.753 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T18:45:19.760 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-09-10T18:45:19.779 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-09-10T18:45:19.779 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-09-10T18:45:19.837 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="CentOS Stream" 2024-09-10T18:45:19.837 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="9" 2024-09-10T18:45:19.837 INFO:teuthology.orchestra.run.smithi119.stdout:ID="centos" 2024-09-10T18:45:19.837 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="rhel fedora" 2024-09-10T18:45:19.837 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="9" 2024-09-10T18:45:19.837 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el9" 2024-09-10T18:45:19.838 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T18:45:19.838 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2024-09-10T18:45:19.838 INFO:teuthology.orchestra.run.smithi119.stdout:LOGO="fedora-logo-icon" 2024-09-10T18:45:19.838 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T18:45:19.838 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://centos.org/" 2024-09-10T18:45:19.838 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T18:45:19.838 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T18:45:19.838 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T18:45:19.839 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-09-10T18:45:19.861 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-09-10T18:45:19.880 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-09-10T18:45:19.880 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-09-10T18:45:19.938 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-09-10T18:45:19.938 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="9" 2024-09-10T18:45:19.939 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-09-10T18:45:19.939 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-09-10T18:45:19.939 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="9" 2024-09-10T18:45:19.939 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el9" 2024-09-10T18:45:19.939 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T18:45:19.939 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-09-10T18:45:19.939 INFO:teuthology.orchestra.run.smithi194.stdout:LOGO="fedora-logo-icon" 2024-09-10T18:45:19.939 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T18:45:19.939 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-09-10T18:45:19.939 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T18:45:19.939 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T18:45:19.939 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T18:45:19.940 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-09-10T18:45:19.961 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T18:45:19.971 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T18:45:19.977 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T18:45:19.978 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-09-10T18:45:19.980 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-09-10T18:45:19.997 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T18:45:20.004 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T18:45:20.004 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-09-10T18:45:20.038 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-09-10T18:45:20.145 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T18:45:20.195 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T18:45:20.210 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-10T18:45:20.210 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-10T18:45:20.210 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T18:45:20.211 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T18:45:20.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:45:20.229 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-09-10T18:45:20.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:45:20.230 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-09-10T18:45:20.287 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-503.el9.x86_64 2024-09-10T18:45:20.288 INFO:teuthology.task.kernel:Running kernel on smithi119: 5.14.0-503.el9.x86_64 2024-09-10T18:45:20.288 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum install -y kernel 2024-09-10T18:45:20.289 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-503.el9.x86_64 2024-09-10T18:45:20.289 INFO:teuthology.task.kernel:Running kernel on smithi194: 5.14.0-503.el9.x86_64 2024-09-10T18:45:20.289 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum install -y kernel 2024-09-10T18:45:22.584 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-10T18:45:22.625 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-10T18:45:24.501 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-09-10T18:45:28.734 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.5 MB 00:00 2024-09-10T18:45:30.352 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 19 kB 00:00 2024-09-10T18:45:31.098 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 36 MB/s | 23 MB 00:00 2024-09-10T18:45:36.495 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-10T18:45:38.528 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T18:45:38.528 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T18:45:38.626 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:45:38.633 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:45:38.633 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-09-10T18:45:38.634 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:45:38.634 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:45:38.634 INFO:teuthology.orchestra.run.smithi194.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T18:45:38.634 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-10T18:45:38.634 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T18:45:38.634 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T18:45:38.634 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T18:45:38.634 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:45:38.634 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:45:38.634 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:45:38.634 INFO:teuthology.orchestra.run.smithi194.stdout:Install 4 Packages 2024-09-10T18:45:38.634 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:45:38.635 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 79 M 2024-09-10T18:45:38.635 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 126 M 2024-09-10T18:45:38.635 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:45:38.829 INFO:teuthology.orchestra.run.smithi194.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 243 kB/s | 23 kB 00:00 2024-09-10T18:45:39.205 INFO:teuthology.orchestra.run.smithi194.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-09-10T18:45:39.597 INFO:teuthology.orchestra.run.smithi194.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 40 MB/s | 35 MB 00:00 2024-09-10T18:45:39.939 INFO:teuthology.orchestra.run.smithi194.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 26 MB/s | 29 MB 00:01 2024-09-10T18:45:39.939 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:45:39.939 INFO:teuthology.orchestra.run.smithi194.stdout:Total 60 MB/s | 79 MB 00:01 2024-09-10T18:45:40.626 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:45:40.961 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:45:40.961 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:45:41.132 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 20 MB 00:17 2024-09-10T18:45:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:45:41.908 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:45:44.343 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:45:44.982 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T18:45:45.088 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:45:47.886 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:45:48.007 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:45:48.866 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.5 MB 00:03 2024-09-10T18:45:50.591 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 83 kB/s | 19 kB 00:00 2024-09-10T18:45:51.496 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-10T18:45:52.817 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:45:52.941 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:45:56.943 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-09-10T18:45:57.779 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:45:58.958 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T18:45:58.958 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T18:45:59.035 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:45:59.046 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:45:59.046 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-09-10T18:45:59.046 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:45:59.046 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:45:59.046 INFO:teuthology.orchestra.run.smithi119.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T18:45:59.047 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-10T18:45:59.047 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T18:45:59.047 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T18:45:59.047 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T18:45:59.047 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:45:59.047 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:45:59.047 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:45:59.047 INFO:teuthology.orchestra.run.smithi119.stdout:Install 4 Packages 2024-09-10T18:45:59.047 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:45:59.047 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 79 M 2024-09-10T18:45:59.047 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 126 M 2024-09-10T18:45:59.047 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:45:59.703 INFO:teuthology.orchestra.run.smithi119.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 42 kB/s | 23 kB 00:00 2024-09-10T18:46:00.587 INFO:teuthology.orchestra.run.smithi119.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-10T18:46:01.272 INFO:teuthology.orchestra.run.smithi119.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-10T18:46:02.264 INFO:teuthology.orchestra.run.smithi119.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 11 MB/s | 29 MB 00:02 2024-09-10T18:46:02.264 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:46:02.265 INFO:teuthology.orchestra.run.smithi119.stdout:Total 24 MB/s | 79 MB 00:03 2024-09-10T18:46:02.952 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:46:03.301 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:46:03.301 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:46:04.245 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:46:04.245 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:46:06.714 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:46:07.374 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T18:46:07.456 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:46:10.255 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:46:10.400 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:46:15.235 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:46:15.360 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:46:20.222 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:46:36.026 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:46:36.065 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:46:36.387 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:46:36.387 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T18:46:36.387 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:46:36.387 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:46:37.258 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:46:37.258 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:46:37.258 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:46:37.259 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T18:46:37.259 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T18:46:37.259 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T18:46:37.259 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T18:46:37.259 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:46:37.259 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:46:37.618 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T18:46:37.698 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T18:46:37.698 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T18:46:37.698 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T18:46:37.699 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T18:46:37.699 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T18:46:37.699 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-10T18:46:37.699 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi194.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T18:46:37.699 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T18:46:37.757 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T18:46:37.758 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T18:46:37.758 DEBUG:teuthology.orchestra.run.smithi194:> sudo rpm -qi grub2-tools 2024-09-10T18:46:37.790 INFO:teuthology.orchestra.run.smithi194.stdout:Name : grub2-tools 2024-09-10T18:46:37.790 INFO:teuthology.orchestra.run.smithi194.stdout:Epoch : 1 2024-09-10T18:46:37.790 INFO:teuthology.orchestra.run.smithi194.stdout:Version : 2.06 2024-09-10T18:46:37.790 INFO:teuthology.orchestra.run.smithi194.stdout:Release : 82.el9 2024-09-10T18:46:37.790 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: x86_64 2024-09-10T18:46:37.790 INFO:teuthology.orchestra.run.smithi194.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T18:46:37.790 INFO:teuthology.orchestra.run.smithi194.stdout:Group : Unspecified 2024-09-10T18:46:37.791 INFO:teuthology.orchestra.run.smithi194.stdout:Size : 8274670 2024-09-10T18:46:37.791 INFO:teuthology.orchestra.run.smithi194.stdout:License : GPLv3+ 2024-09-10T18:46:37.791 INFO:teuthology.orchestra.run.smithi194.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T18:46:37.791 INFO:teuthology.orchestra.run.smithi194.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T18:46:37.791 INFO:teuthology.orchestra.run.smithi194.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T18:46:37.791 INFO:teuthology.orchestra.run.smithi194.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T18:46:37.791 INFO:teuthology.orchestra.run.smithi194.stdout:Packager : builder@centos.org 2024-09-10T18:46:37.791 INFO:teuthology.orchestra.run.smithi194.stdout:Vendor : CentOS 2024-09-10T18:46:37.791 INFO:teuthology.orchestra.run.smithi194.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T18:46:37.791 INFO:teuthology.orchestra.run.smithi194.stdout:Summary : Support tools for GRUB. 2024-09-10T18:46:37.791 INFO:teuthology.orchestra.run.smithi194.stdout:Description : 2024-09-10T18:46:37.791 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:46:37.791 INFO:teuthology.orchestra.run.smithi194.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T18:46:37.791 INFO:teuthology.orchestra.run.smithi194.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T18:46:37.792 INFO:teuthology.orchestra.run.smithi194.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T18:46:37.792 INFO:teuthology.orchestra.run.smithi194.stdout:hardware devices. 2024-09-10T18:46:37.792 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:46:37.792 INFO:teuthology.orchestra.run.smithi194.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T18:46:37.793 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T18:46:37.793 INFO:teuthology.task.kernel:Updating grub on smithi194 to boot 5.14.0-505.el9.x86_64 2024-09-10T18:46:37.793 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T18:46:38.423 INFO:teuthology.orchestra.run.smithi194.stderr:Generating grub configuration file ... 2024-09-10T18:46:39.740 INFO:teuthology.orchestra.run.smithi194.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T18:46:39.768 INFO:teuthology.orchestra.run.smithi194.stderr:done 2024-09-10T18:46:39.770 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-09-10T18:46:39.786 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.ZxEQnDLAVo 2024-09-10T18:46:39.786 DEBUG:teuthology.orchestra.run.smithi194:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZxEQnDLAVo 2024-09-10T18:46:39.853 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.ZxEQnDLAVo 2024-09-10T18:46:39.988 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.ZxEQnDLAVo is 6KB 2024-09-10T18:46:40.036 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.ZxEQnDLAVo 2024-09-10T18:46:40.051 DEBUG:teuthology.orchestra.run.smithi194:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T18:46:40.113 INFO:teuthology.orchestra.run.smithi194.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T18:46:40.113 INFO:teuthology.orchestra.run.smithi194.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T18:46:40.113 INFO:teuthology.orchestra.run.smithi194.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T18:46:40.114 INFO:teuthology.orchestra.run.smithi194.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T18:46:40.115 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T18:46:40.281 DEBUG:teuthology.orchestra.run.smithi194:> sudo shutdown -r now 2024-09-10T18:46:58.609 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:46:58.653 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:46:59.043 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:46:59.043 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T18:46:59.043 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:46:59.043 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:46:59.886 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:46:59.886 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:46:59.886 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:46:59.886 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T18:46:59.886 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T18:46:59.886 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T18:46:59.887 INFO:teuthology.orchestra.run.smithi119.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T18:46:59.887 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:46:59.887 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:47:00.275 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T18:47:00.413 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T18:47:00.413 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T18:47:00.413 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T18:47:00.413 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T18:47:00.414 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T18:47:00.414 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-10T18:47:00.414 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi119.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T18:47:00.414 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T18:47:00.473 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T18:47:00.473 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T18:47:00.473 DEBUG:teuthology.orchestra.run.smithi119:> sudo rpm -qi grub2-tools 2024-09-10T18:47:00.512 INFO:teuthology.orchestra.run.smithi119.stdout:Name : grub2-tools 2024-09-10T18:47:00.512 INFO:teuthology.orchestra.run.smithi119.stdout:Epoch : 1 2024-09-10T18:47:00.512 INFO:teuthology.orchestra.run.smithi119.stdout:Version : 2.06 2024-09-10T18:47:00.512 INFO:teuthology.orchestra.run.smithi119.stdout:Release : 82.el9 2024-09-10T18:47:00.512 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: x86_64 2024-09-10T18:47:00.512 INFO:teuthology.orchestra.run.smithi119.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T18:47:00.512 INFO:teuthology.orchestra.run.smithi119.stdout:Group : Unspecified 2024-09-10T18:47:00.513 INFO:teuthology.orchestra.run.smithi119.stdout:Size : 8274670 2024-09-10T18:47:00.513 INFO:teuthology.orchestra.run.smithi119.stdout:License : GPLv3+ 2024-09-10T18:47:00.513 INFO:teuthology.orchestra.run.smithi119.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T18:47:00.513 INFO:teuthology.orchestra.run.smithi119.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T18:47:00.513 INFO:teuthology.orchestra.run.smithi119.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T18:47:00.513 INFO:teuthology.orchestra.run.smithi119.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T18:47:00.513 INFO:teuthology.orchestra.run.smithi119.stdout:Packager : builder@centos.org 2024-09-10T18:47:00.513 INFO:teuthology.orchestra.run.smithi119.stdout:Vendor : CentOS 2024-09-10T18:47:00.513 INFO:teuthology.orchestra.run.smithi119.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T18:47:00.513 INFO:teuthology.orchestra.run.smithi119.stdout:Summary : Support tools for GRUB. 2024-09-10T18:47:00.513 INFO:teuthology.orchestra.run.smithi119.stdout:Description : 2024-09-10T18:47:00.513 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:47:00.513 INFO:teuthology.orchestra.run.smithi119.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T18:47:00.513 INFO:teuthology.orchestra.run.smithi119.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T18:47:00.514 INFO:teuthology.orchestra.run.smithi119.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T18:47:00.514 INFO:teuthology.orchestra.run.smithi119.stdout:hardware devices. 2024-09-10T18:47:00.514 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:47:00.514 INFO:teuthology.orchestra.run.smithi119.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T18:47:00.515 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T18:47:00.515 INFO:teuthology.task.kernel:Updating grub on smithi119 to boot 5.14.0-505.el9.x86_64 2024-09-10T18:47:00.515 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T18:47:01.152 INFO:teuthology.orchestra.run.smithi119.stderr:Generating grub configuration file ... 2024-09-10T18:47:02.401 INFO:teuthology.orchestra.run.smithi119.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T18:47:02.431 INFO:teuthology.orchestra.run.smithi119.stderr:done 2024-09-10T18:47:02.433 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-09-10T18:47:02.449 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.hGgZy01UC1 2024-09-10T18:47:02.449 DEBUG:teuthology.orchestra.run.smithi119:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.hGgZy01UC1 2024-09-10T18:47:02.515 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.hGgZy01UC1 2024-09-10T18:47:02.653 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.hGgZy01UC1 is 6KB 2024-09-10T18:47:02.702 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.hGgZy01UC1 2024-09-10T18:47:02.717 DEBUG:teuthology.orchestra.run.smithi119:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T18:47:02.789 INFO:teuthology.orchestra.run.smithi119.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T18:47:02.789 INFO:teuthology.orchestra.run.smithi119.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T18:47:02.789 INFO:teuthology.orchestra.run.smithi119.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T18:47:02.790 INFO:teuthology.orchestra.run.smithi119.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T18:47:02.791 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T18:47:02.943 DEBUG:teuthology.orchestra.run.smithi119:> sudo shutdown -r now 2024-09-10T18:47:10.291 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-10T18:47:10.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-10T18:47:10.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:47:28.829 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-09-10T18:47:32.949 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-10T18:47:32.949 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-10T18:47:32.950 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:47:37.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-10T18:47:37.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:47:40.893 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-09-10T18:47:51.362 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2024-09-10T18:47:52.896 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-10T18:47:52.897 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:48:00.370 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-10T18:48:00.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:48:11.329 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-09-10T18:48:26.343 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-10T18:48:26.344 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:48:26.727 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-09-10T18:48:27.145 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-10T18:48:27.145 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-10T18:48:27.145 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-09-10T18:48:27.160 INFO:teuthology.orchestra.run.smithi194.stdout:5.14.0-505.el9.x86_64 2024-09-10T18:48:27.160 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T18:48:27.160 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T18:48:27.160 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T18:48:28.161 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-10T18:48:28.161 DEBUG:teuthology.orchestra.run.smithi194:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T18:48:28.439 INFO:teuthology.orchestra.run.smithi194.stdout:ttyS1 2024-09-10T18:48:28.466 DEBUG:teuthology.parallel:result is None 2024-09-10T18:49:00.374 DEBUG:teuthology.orchestra.remote:timed out 2024-09-10T18:49:12.375 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-10T18:49:12.376 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:49:12.991 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-09-10T18:49:13.345 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-10T18:49:13.345 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-10T18:49:13.346 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-09-10T18:49:13.362 INFO:teuthology.orchestra.run.smithi119.stdout:5.14.0-505.el9.x86_64 2024-09-10T18:49:13.362 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T18:49:13.362 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T18:49:13.362 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T18:49:14.364 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-10T18:49:14.364 DEBUG:teuthology.orchestra.run.smithi119:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T18:49:14.707 INFO:teuthology.orchestra.run.smithi119.stdout:ttyS1 2024-09-10T18:49:14.731 DEBUG:teuthology.parallel:result is None 2024-09-10T18:49:14.731 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T18:49:14.742 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T18:49:14.742 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T18:49:14.745 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T18:49:14.764 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T18:49:14.779 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T18:49:14.787 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T18:49:14.787 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T18:49:14.803 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T18:49:14.857 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T18:49:14.864 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T18:49:14.865 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T18:49:14.867 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T18:49:14.898 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:49:14.899 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:49:14.909 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:49:14.910 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:49:14.912 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T18:49:14.919 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T18:49:14.919 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T18:49:14.954 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T18:49:14.981 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T18:49:14.990 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T18:49:14.991 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T18:49:15.024 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T18:49:15.041 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T18:49:15.123 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T18:49:15.172 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T18:49:15.255 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T18:49:15.295 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T18:49:15.296 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T18:49:15.361 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T18:49:15.406 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T18:49:15.468 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T18:49:15.508 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T18:49:15.553 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-10T18:49:15.553 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T18:49:15.620 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-09-10T18:49:15.622 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-09-10T18:49:15.672 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T18:49:15.712 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T18:49:16.114 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T18:49:16.122 INFO:teuthology.task.internal:Starting timer... 2024-09-10T18:49:16.122 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T18:49:16.138 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T18:49:16.148 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-10T18:49:16.149 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2024-09-10T18:49:16.213 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: 2024-09-10T18:49:16.214 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-09-10T18:49:16.264 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-09-10T18:49:16.265 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T18:49:16.266 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2024-09-10T18:49:16.288 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2024-09-10T18:49:16.289 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-09-10T18:49:16.313 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-09-10T18:49:16.313 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi119.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-09-10T18:49:16.314 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T18:49:16.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:49:16.355 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T18:49:16.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:49:16.384 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T18:49:16.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2024-09-10T18:49:16.427 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-09-10T18:49:16.465 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T18:49:16.475 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T18:49:16.487 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T18:49:16.488 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi119.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T18:52:59.536 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi119.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-09-10T18:52:59.537 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-10T18:52:59.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:52:59.614 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-09-10T18:52:59.700 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-10T18:52:59.701 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-10T18:52:59.701 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:52:59.775 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-09-10T18:52:59.856 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-09-10T18:52:59.856 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T18:52:59.867 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T18:52:59.867 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T18:52:59.868 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T18:52:59.870 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T18:52:59.870 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T18:52:59.897 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T18:52:59.911 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T18:52:59.925 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T18:52:59.934 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2024-09-10T18:52:59.940 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T18:52:59.945 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2024-09-10T18:52:59.959 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T18:52:59.965 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-09-10T18:52:59.973 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T18:52:59.975 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-09-10T18:52:59.990 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T18:53:00.003 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T18:53:00.030 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-09-10T18:53:00.033 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T18:53:00.033 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-09-10T18:53:00.033 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:53:00.034 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:53:00.034 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:53:00.034 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:53:00.061 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-09-10T18:53:00.064 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T18:53:00.064 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-09-10T18:53:00.065 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:53:00.065 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:53:00.065 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:53:00.065 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:53:00.066 INFO:teuthology.run_tasks:Running task pexec... 2024-09-10T18:53:00.080 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-10T18:53:00.081 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi119.front.sepia.ceph.com 2024-09-10T18:53:00.081 DEBUG:teuthology.orchestra.run.smithi119:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-10T18:53:00.082 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi194.front.sepia.ceph.com 2024-09-10T18:53:00.082 DEBUG:teuthology.orchestra.run.smithi194:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-10T18:53:00.431 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:53:00.443 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:53:00.443 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-09-10T18:53:00.443 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:53:00.443 INFO:teuthology.orchestra.run.smithi119.stdout:Removing: 2024-09-10T18:53:00.443 INFO:teuthology.orchestra.run.smithi119.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-10T18:53:00.443 INFO:teuthology.orchestra.run.smithi119.stdout:Removing unused dependencies: 2024-09-10T18:53:00.443 INFO:teuthology.orchestra.run.smithi119.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-10T18:53:00.443 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:53:00.444 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:53:00.444 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:53:00.444 INFO:teuthology.orchestra.run.smithi119.stdout:Remove 2 Packages 2024-09-10T18:53:00.444 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:53:00.444 INFO:teuthology.orchestra.run.smithi119.stdout:Freed space: 5.5 M 2024-09-10T18:53:00.444 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:53:00.447 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:53:00.447 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:53:00.456 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:53:00.458 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:53:00.458 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-09-10T18:53:00.458 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:53:00.458 INFO:teuthology.orchestra.run.smithi194.stdout:Removing: 2024-09-10T18:53:00.458 INFO:teuthology.orchestra.run.smithi194.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-10T18:53:00.458 INFO:teuthology.orchestra.run.smithi194.stdout:Removing unused dependencies: 2024-09-10T18:53:00.459 INFO:teuthology.orchestra.run.smithi194.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-10T18:53:00.459 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:00.459 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:53:00.459 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:53:00.459 INFO:teuthology.orchestra.run.smithi194.stdout:Remove 2 Packages 2024-09-10T18:53:00.459 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:00.459 INFO:teuthology.orchestra.run.smithi194.stdout:Freed space: 5.5 M 2024-09-10T18:53:00.459 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:53:00.463 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:53:00.463 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:53:00.516 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:53:00.517 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:53:00.526 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:53:00.527 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:53:00.649 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:53:00.663 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:53:00.729 INFO:teuthology.orchestra.run.smithi119.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-10T18:53:00.745 INFO:teuthology.orchestra.run.smithi194.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-10T18:53:00.781 INFO:teuthology.orchestra.run.smithi119.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-10T18:53:00.799 INFO:teuthology.orchestra.run.smithi194.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-10T18:53:01.321 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-10T18:53:01.321 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-10T18:53:01.357 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-10T18:53:01.357 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-10T18:53:01.558 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-10T18:53:01.558 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:53:01.558 INFO:teuthology.orchestra.run.smithi119.stdout:Removed: 2024-09-10T18:53:01.558 INFO:teuthology.orchestra.run.smithi119.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-10T18:53:01.558 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:53:01.558 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:53:01.599 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-10T18:53:01.600 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:01.600 INFO:teuthology.orchestra.run.smithi194.stdout:Removed: 2024-09-10T18:53:01.600 INFO:teuthology.orchestra.run.smithi194.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-10T18:53:01.600 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:01.600 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:53:02.343 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:49 ago on Tue 10 Sep 2024 06:52:13 PM UTC. 2024-09-10T18:53:02.445 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:49 ago on Tue 10 Sep 2024 06:52:13 PM UTC. 2024-09-10T18:53:02.481 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:53:02.481 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:53:02.481 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-10T18:53:02.481 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:53:02.481 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:53:02.481 INFO:teuthology.orchestra.run.smithi119.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-10T18:53:02.481 INFO:teuthology.orchestra.run.smithi119.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-10T18:53:02.482 INFO:teuthology.orchestra.run.smithi119.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-10T18:53:02.482 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-10T18:53:02.482 INFO:teuthology.orchestra.run.smithi119.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-10T18:53:02.482 INFO:teuthology.orchestra.run.smithi119.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-10T18:53:02.482 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-10T18:53:02.482 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-10T18:53:02.482 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:53:02.482 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:53:02.482 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:53:02.482 INFO:teuthology.orchestra.run.smithi119.stdout:Install 7 Packages 2024-09-10T18:53:02.482 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:53:02.536 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 5.2 M 2024-09-10T18:53:02.537 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 19 M 2024-09-10T18:53:02.537 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:53:02.588 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:53:02.589 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:53:02.589 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-10T18:53:02.589 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:53:02.589 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:53:02.589 INFO:teuthology.orchestra.run.smithi194.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-10T18:53:02.589 INFO:teuthology.orchestra.run.smithi194.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-10T18:53:02.589 INFO:teuthology.orchestra.run.smithi194.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-10T18:53:02.589 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-10T18:53:02.590 INFO:teuthology.orchestra.run.smithi194.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-10T18:53:02.590 INFO:teuthology.orchestra.run.smithi194.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-10T18:53:02.590 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-10T18:53:02.590 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-10T18:53:02.590 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:02.590 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:53:02.590 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:53:02.590 INFO:teuthology.orchestra.run.smithi194.stdout:Install 7 Packages 2024-09-10T18:53:02.590 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:02.616 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 5.2 M 2024-09-10T18:53:02.616 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 19 M 2024-09-10T18:53:02.617 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:53:02.829 INFO:teuthology.orchestra.run.smithi119.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 303 kB/s | 44 kB 00:00 2024-09-10T18:53:02.854 INFO:teuthology.orchestra.run.smithi119.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 616 kB/s | 106 kB 00:00 2024-09-10T18:53:02.913 INFO:teuthology.orchestra.run.smithi119.stdout:(3/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-10T18:53:02.938 INFO:teuthology.orchestra.run.smithi119.stdout:(4/7): python3-configshell-1.1.30-1.el9.noarch. 659 kB/s | 72 kB 00:00 2024-09-10T18:53:02.980 INFO:teuthology.orchestra.run.smithi119.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.0 MB/s | 906 kB 00:00 2024-09-10T18:53:02.996 INFO:teuthology.orchestra.run.smithi194.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 213 kB/s | 44 kB 00:00 2024-09-10T18:53:03.072 INFO:teuthology.orchestra.run.smithi119.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 5.1 MB/s | 837 kB 00:00 2024-09-10T18:53:03.079 INFO:teuthology.orchestra.run.smithi194.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 364 kB/s | 106 kB 00:00 2024-09-10T18:53:03.113 INFO:teuthology.orchestra.run.smithi194.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 614 kB/s | 72 kB 00:00 2024-09-10T18:53:03.255 INFO:teuthology.orchestra.run.smithi194.stdout:(4/7): python3-kmod-0.9-32.el9.x86_64.rpm 480 kB/s | 84 kB 00:00 2024-09-10T18:53:03.480 INFO:teuthology.orchestra.run.smithi194.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 1.3 MB/s | 906 kB 00:00 2024-09-10T18:53:03.631 INFO:teuthology.orchestra.run.smithi194.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 1.6 MB/s | 837 kB 00:00 2024-09-10T18:53:04.397 INFO:teuthology.orchestra.run.smithi119.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 2.2 MB/s | 3.2 MB 00:01 2024-09-10T18:53:04.398 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:53:04.398 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.8 MB/s | 5.2 MB 00:01 2024-09-10T18:53:04.473 INFO:teuthology.orchestra.run.smithi194.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 2.6 MB/s | 3.2 MB 00:01 2024-09-10T18:53:04.473 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:53:04.473 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.8 MB/s | 5.2 MB 00:01 2024-09-10T18:53:04.517 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:53:04.532 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:53:04.532 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:53:04.584 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:53:04.599 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:53:04.599 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:53:04.855 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:53:04.855 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:53:04.903 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:53:04.903 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:53:05.342 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:53:05.376 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:53:05.433 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-10T18:53:05.466 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-10T18:53:05.475 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-10T18:53:05.501 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-10T18:53:05.505 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-10T18:53:05.529 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-10T18:53:05.712 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-10T18:53:05.742 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-10T18:53:05.744 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-10T18:53:05.777 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-10T18:53:06.188 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-10T18:53:06.221 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-10T18:53:06.222 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-10T18:53:06.256 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-10T18:53:06.333 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-10T18:53:06.362 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-10T18:53:06.409 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-10T18:53:06.425 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-10T18:53:07.350 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-10T18:53:07.350 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-10T18:53:07.351 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-10T18:53:07.351 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-10T18:53:07.351 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-10T18:53:07.351 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-10T18:53:07.351 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-10T18:53:07.427 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-10T18:53:07.427 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-10T18:53:07.427 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-10T18:53:07.427 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-10T18:53:07.427 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-10T18:53:07.427 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-10T18:53:07.427 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-10T18:53:07.711 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-10T18:53:07.711 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:53:07.711 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:53:07.711 INFO:teuthology.orchestra.run.smithi119.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-10T18:53:07.711 INFO:teuthology.orchestra.run.smithi119.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-10T18:53:07.711 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-10T18:53:07.711 INFO:teuthology.orchestra.run.smithi119.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-10T18:53:07.711 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:53:07.711 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:53:07.832 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-10T18:53:07.832 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:07.832 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:53:07.832 INFO:teuthology.orchestra.run.smithi194.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-10T18:53:07.832 INFO:teuthology.orchestra.run.smithi194.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-10T18:53:07.832 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-10T18:53:07.832 INFO:teuthology.orchestra.run.smithi194.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-10T18:53:07.832 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:07.832 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:53:08.098 DEBUG:teuthology.parallel:result is None 2024-09-10T18:53:08.239 DEBUG:teuthology.parallel:result is None 2024-09-10T18:53:08.239 INFO:teuthology.run_tasks:Running task install... 2024-09-10T18:53:08.248 DEBUG:teuthology.task.install:project ceph 2024-09-10T18:53:08.248 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588'}, 'extra_packages': ['cephadm']} 2024-09-10T18:53:08.248 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588'} 2024-09-10T18:53:08.248 INFO:teuthology.task.install:Using flavor: default 2024-09-10T18:53:08.254 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T18:53:08.254 INFO:teuthology.task.install:extra packages: [] 2024-09-10T18:53:08.254 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588', 'tag': None, 'wait_for_package': False} 2024-09-10T18:53:08.255 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:53:08.257 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588', 'tag': None, 'wait_for_package': False} 2024-09-10T18:53:08.257 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:53:08.397 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/ 2024-09-10T18:53:08.397 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1606.g5be57395 2024-09-10T18:53:08.404 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/ 2024-09-10T18:53:08.404 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1606.g5be57395 2024-09-10T18:53:08.507 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T18:53:08.507 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T18:53:08.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T18:53:08.516 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T18:53:08.516 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-10T18:53:08.516 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T18:53:08.534 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T18:53:08.534 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5be57395ea235535910512ae73ba8e7ff25fe588/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T18:53:08.542 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T18:53:08.543 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5be57395ea235535910512ae73ba8e7ff25fe588/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T18:53:08.600 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T18:53:08.619 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T18:53:08.675 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T18:53:08.694 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T18:53:08.739 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2024-09-10T18:53:08.741 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2024-09-10T18:53:08.758 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-09-10T18:53:08.760 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-09-10T18:53:08.964 INFO:teuthology.orchestra.run.smithi119.stdout:45 files removed 2024-09-10T18:53:08.991 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw 2024-09-10T18:53:08.995 INFO:teuthology.orchestra.run.smithi194.stdout:45 files removed 2024-09-10T18:53:09.022 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw 2024-09-10T18:53:09.545 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 292 kB/s | 77 kB 00:00 2024-09-10T18:53:09.570 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 296 kB/s | 77 kB 00:00 2024-09-10T18:53:09.829 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 45 kB/s | 11 kB 00:00 2024-09-10T18:53:09.862 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-09-10T18:53:10.096 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 7.8 kB/s | 1.8 kB 00:00 2024-09-10T18:53:10.138 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-10T18:53:12.513 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 3.5 MB/s | 8.2 MB 00:02 2024-09-10T18:53:12.638 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - BaseOS 3.3 MB/s | 8.2 MB 00:02 2024-09-10T18:53:14.464 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 24 MB/s | 20 MB 00:00 2024-09-10T18:53:20.940 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.5 MB 00:02 2024-09-10T18:53:22.105 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - AppStream 2.4 MB/s | 20 MB 00:08 2024-09-10T18:53:22.432 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 19 kB 00:00 2024-09-10T18:53:23.294 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-10T18:53:26.322 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - CRB 17 MB/s | 6.5 MB 00:00 2024-09-10T18:53:27.848 INFO:teuthology.orchestra.run.smithi119.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 19 kB 00:00 2024-09-10T18:53:28.635 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-10T18:53:28.700 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 36 kB/s | 1.7 kB 00:00 2024-09-10T18:53:31.082 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:53:31.084 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:53:31.084 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-10T18:53:31.084 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:53:31.084 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:53:31.084 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:17.2.7-1606.g5be57395.el9 ceph 14 M 2024-09-10T18:53:31.084 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-09-10T18:53:31.085 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 3.1 M 2024-09-10T18:53:31.085 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.9 M 2024-09-10T18:53:31.085 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-10T18:53:31.085 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:17.2.7-1606.g5be57395.el9 ceph 5.0 M 2024-09-10T18:53:31.085 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:17.2.7-1606.g5be57395.el9 ceph 22 M 2024-09-10T18:53:31.085 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:17.2.7-1606.g5be57395.el9 ceph 25 k 2024-09-10T18:53:31.085 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T18:53:31.085 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 637 k 2024-09-10T18:53:31.085 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T18:53:31.085 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T18:53:31.085 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T18:53:31.085 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 458 k 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 4.9 M 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 45 k 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common x86_64 2:17.2.7-1606.g5be57395.el9 ceph 120 k 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:17.2.7-1606.g5be57395.el9 ceph 156 k 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:17.2.7-1606.g5be57395.el9 ceph 320 k 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 295 k 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:17.2.7-1606.g5be57395.el9 ceph 99 k 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:53:31.086 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:53:31.087 INFO:teuthology.orchestra.run.smithi194.stdout:Install 22 Packages 2024-09-10T18:53:31.087 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 2 Packages 2024-09-10T18:53:31.087 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:31.087 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 58 M 2024-09-10T18:53:31.087 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:53:32.068 INFO:teuthology.orchestra.run.smithi194.stdout:(1/24): ceph-base-17.2.7-1606.g5be57395.el9.x86 9.1 MB/s | 5.0 MB 00:00 2024-09-10T18:53:32.118 INFO:teuthology.orchestra.run.smithi194.stdout:(2/24): ceph-selinux-17.2.7-1606.g5be57395.el9. 505 kB/s | 25 kB 00:00 2024-09-10T18:53:32.202 INFO:teuthology.orchestra.run.smithi194.stdout:(3/24): libcephfs2-17.2.7-1606.g5be57395.el9.x8 7.5 MB/s | 637 kB 00:00 2024-09-10T18:53:32.277 INFO:teuthology.orchestra.run.smithi194.stdout:(4/24): libradosstriper1-17.2.7-1606.g5be57395. 6.0 MB/s | 458 kB 00:00 2024-09-10T18:53:32.644 INFO:teuthology.orchestra.run.smithi194.stdout:(5/24): ceph-radosgw-17.2.7-1606.g5be57395.el9. 13 MB/s | 14 MB 00:01 2024-09-10T18:53:32.861 INFO:teuthology.orchestra.run.smithi194.stdout:(6/24): ceph-common-17.2.7-1606.g5be57395.el9.x 17 MB/s | 22 MB 00:01 2024-09-10T18:53:32.886 INFO:teuthology.orchestra.run.smithi194.stdout:(7/24): python3-ceph-argparse-17.2.7-1606.g5be5 187 kB/s | 45 kB 00:00 2024-09-10T18:53:32.911 INFO:teuthology.orchestra.run.smithi194.stdout:(8/24): python3-ceph-common-17.2.7-1606.g5be573 2.4 MB/s | 120 kB 00:00 2024-09-10T18:53:32.978 INFO:teuthology.orchestra.run.smithi194.stdout:(9/24): librgw2-17.2.7-1606.g5be57395.el9.x86_6 7.0 MB/s | 4.9 MB 00:00 2024-09-10T18:53:33.004 INFO:teuthology.orchestra.run.smithi194.stdout:(10/24): python3-cephfs-17.2.7-1606.g5be57395.e 1.3 MB/s | 156 kB 00:00 2024-09-10T18:53:33.037 INFO:teuthology.orchestra.run.smithi194.stdout:(11/24): python3-rados-17.2.7-1606.g5be57395.el 2.5 MB/s | 320 kB 00:00 2024-09-10T18:53:33.062 INFO:teuthology.orchestra.run.smithi194.stdout:(12/24): python3-rbd-17.2.7-1606.g5be57395.el9. 3.4 MB/s | 295 kB 00:00 2024-09-10T18:53:33.087 INFO:teuthology.orchestra.run.smithi194.stdout:(13/24): python3-rgw-17.2.7-1606.g5be57395.el9. 1.2 MB/s | 99 kB 00:00 2024-09-10T18:53:33.438 INFO:teuthology.orchestra.run.smithi194.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 121 kB/s | 45 kB 00:00 2024-09-10T18:53:33.613 INFO:teuthology.orchestra.run.smithi194.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 278 kB/s | 160 kB 00:00 2024-09-10T18:53:33.738 INFO:teuthology.orchestra.run.smithi194.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 974 kB/s | 292 kB 00:00 2024-09-10T18:53:33.764 INFO:teuthology.orchestra.run.smithi194.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 281 kB/s | 42 kB 00:00 2024-09-10T18:53:33.822 INFO:teuthology.orchestra.run.smithi194.stdout:(18/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.6 MB/s | 309 kB 00:00 2024-09-10T18:53:33.848 INFO:teuthology.orchestra.run.smithi194.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 582 kB/s | 49 kB 00:00 2024-09-10T18:53:33.872 INFO:teuthology.orchestra.run.smithi194.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-10T18:53:33.906 INFO:teuthology.orchestra.run.smithi194.stdout:(21/24): librdkafka-1.6.1-102.el9.x86_64.rpm 809 kB/s | 662 kB 00:00 2024-09-10T18:53:33.957 INFO:teuthology.orchestra.run.smithi194.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-10T18:53:34.090 INFO:teuthology.orchestra.run.smithi194.stdout:(23/24): librados2-17.2.7-1606.g5be57395.el9.x8 14 MB/s | 3.1 MB 00:00 2024-09-10T18:53:34.116 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-09-10T18:53:34.148 INFO:teuthology.orchestra.run.smithi194.stdout:(24/24): librbd1-17.2.7-1606.g5be57395.el9.x86_ 12 MB/s | 2.9 MB 00:00 2024-09-10T18:53:34.152 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:53:34.153 INFO:teuthology.orchestra.run.smithi194.stdout:Total 19 MB/s | 58 MB 00:03 2024-09-10T18:53:34.213 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:53:34.244 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:53:34.244 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:53:34.714 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:53:34.714 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:53:35.902 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:53:36.305 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-10T18:53:36.368 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T18:53:36.467 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T18:53:36.498 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:53:36.500 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:53:36.500 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-10T18:53:36.500 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:53:36.500 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:53:36.500 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:17.2.7-1606.g5be57395.el9 ceph 14 M 2024-09-10T18:53:36.500 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2024-09-10T18:53:36.500 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 3.1 M 2024-09-10T18:53:36.500 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.9 M 2024-09-10T18:53:36.501 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-10T18:53:36.501 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:17.2.7-1606.g5be57395.el9 ceph 5.0 M 2024-09-10T18:53:36.501 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:17.2.7-1606.g5be57395.el9 ceph 22 M 2024-09-10T18:53:36.501 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:17.2.7-1606.g5be57395.el9 ceph 25 k 2024-09-10T18:53:36.501 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T18:53:36.501 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 637 k 2024-09-10T18:53:36.501 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T18:53:36.501 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T18:53:36.501 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T18:53:36.501 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 458 k 2024-09-10T18:53:36.501 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T18:53:36.501 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 4.9 M 2024-09-10T18:53:36.502 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T18:53:36.502 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T18:53:36.502 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 45 k 2024-09-10T18:53:36.502 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:17.2.7-1606.g5be57395.el9 ceph 120 k 2024-09-10T18:53:36.502 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:17.2.7-1606.g5be57395.el9 ceph 156 k 2024-09-10T18:53:36.502 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T18:53:36.502 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:17.2.7-1606.g5be57395.el9 ceph 320 k 2024-09-10T18:53:36.502 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 295 k 2024-09-10T18:53:36.502 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:17.2.7-1606.g5be57395.el9 ceph 99 k 2024-09-10T18:53:36.502 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T18:53:36.502 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:53:36.502 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:53:36.503 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:53:36.503 INFO:teuthology.orchestra.run.smithi119.stdout:Install 22 Packages 2024-09-10T18:53:36.503 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 2 Packages 2024-09-10T18:53:36.503 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:53:36.503 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 58 M 2024-09-10T18:53:36.503 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:53:36.555 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 3/26 2024-09-10T18:53:36.586 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T18:53:36.674 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T18:53:36.720 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-10T18:53:36.757 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-10T18:53:37.213 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-10T18:53:37.247 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 8/26 2024-09-10T18:53:37.310 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 8/26 2024-09-10T18:53:37.346 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-10T18:53:37.389 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9. 10/26 2024-09-10T18:53:37.433 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 11/26 2024-09-10T18:53:37.450 INFO:teuthology.orchestra.run.smithi119.stdout:(1/24): ceph-base-17.2.7-1606.g5be57395.el9.x86 10 MB/s | 5.0 MB 00:00 2024-09-10T18:53:37.496 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 12/26 2024-09-10T18:53:37.509 INFO:teuthology.orchestra.run.smithi119.stdout:(2/24): ceph-selinux-17.2.7-1606.g5be57395.el9. 431 kB/s | 25 kB 00:00 2024-09-10T18:53:37.523 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 13/26 2024-09-10T18:53:37.572 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 13/26 2024-09-10T18:53:37.601 INFO:teuthology.orchestra.run.smithi119.stdout:(3/24): libcephfs2-17.2.7-1606.g5be57395.el9.x8 6.8 MB/s | 637 kB 00:00 2024-09-10T18:53:37.612 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-10T18:53:37.648 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-09-10T18:53:37.689 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-10T18:53:37.701 INFO:teuthology.orchestra.run.smithi119.stdout:(4/24): libradosstriper1-17.2.7-1606.g5be57395. 4.5 MB/s | 458 kB 00:00 2024-09-10T18:53:37.995 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-10T18:53:38.027 INFO:teuthology.orchestra.run.smithi119.stdout:(5/24): ceph-radosgw-17.2.7-1606.g5be57395.el9. 14 MB/s | 14 MB 00:01 2024-09-10T18:53:38.044 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 18/26 2024-09-10T18:53:38.069 INFO:teuthology.orchestra.run.smithi119.stdout:(6/24): python3-ceph-argparse-17.2.7-1606.g5be5 1.1 MB/s | 45 kB 00:00 2024-09-10T18:53:38.134 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 18/26 2024-09-10T18:53:38.144 INFO:teuthology.orchestra.run.smithi119.stdout:(7/24): librgw2-17.2.7-1606.g5be57395.el9.x86_6 11 MB/s | 4.9 MB 00:00 2024-09-10T18:53:38.169 INFO:teuthology.orchestra.run.smithi119.stdout:(8/24): python3-ceph-common-17.2.7-1606.g5be573 1.2 MB/s | 120 kB 00:00 2024-09-10T18:53:38.194 INFO:teuthology.orchestra.run.smithi119.stdout:(9/24): python3-cephfs-17.2.7-1606.g5be57395.el 3.1 MB/s | 156 kB 00:00 2024-09-10T18:53:38.215 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 19/26 2024-09-10T18:53:38.220 INFO:teuthology.orchestra.run.smithi119.stdout:(10/24): python3-rados-17.2.7-1606.g5be57395.el 6.2 MB/s | 320 kB 00:00 2024-09-10T18:53:38.246 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x8 20/26 2024-09-10T18:53:38.428 INFO:teuthology.orchestra.run.smithi119.stdout:(11/24): ceph-common-17.2.7-1606.g5be57395.el9. 15 MB/s | 22 MB 00:01 2024-09-10T18:53:38.454 INFO:teuthology.orchestra.run.smithi119.stdout:(12/24): python3-rgw-17.2.7-1606.g5be57395.el9. 425 kB/s | 99 kB 00:00 2024-09-10T18:53:38.487 INFO:teuthology.orchestra.run.smithi119.stdout:(13/24): python3-rbd-17.2.7-1606.g5be57395.el9. 1.0 MB/s | 295 kB 00:00 2024-09-10T18:53:39.021 INFO:teuthology.orchestra.run.smithi119.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 80 kB/s | 45 kB 00:00 2024-09-10T18:53:39.238 INFO:teuthology.orchestra.run.smithi119.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 198 kB/s | 160 kB 00:00 2024-09-10T18:53:39.830 INFO:teuthology.orchestra.run.smithi119.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 493 kB/s | 662 kB 00:01 2024-09-10T18:53:39.855 INFO:teuthology.orchestra.run.smithi119.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 68 kB/s | 42 kB 00:00 2024-09-10T18:53:39.914 INFO:teuthology.orchestra.run.smithi119.stdout:(18/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.6 MB/s | 309 kB 00:00 2024-09-10T18:53:39.930 INFO:teuthology.orchestra.run.smithi119.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 723 kB/s | 49 kB 00:00 2024-09-10T18:53:39.956 INFO:teuthology.orchestra.run.smithi119.stdout:(20/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 313 kB/s | 292 kB 00:00 2024-09-10T18:53:39.981 INFO:teuthology.orchestra.run.smithi119.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-09-10T18:53:40.048 INFO:teuthology.orchestra.run.smithi119.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-10T18:53:40.373 INFO:teuthology.orchestra.run.smithi119.stdout:(23/24): librbd1-17.2.7-1606.g5be57395.el9.x86_ 7.5 MB/s | 2.9 MB 00:00 2024-09-10T18:53:40.432 INFO:teuthology.orchestra.run.smithi119.stdout:(24/24): librados2-17.2.7-1606.g5be57395.el9.x8 6.5 MB/s | 3.1 MB 00:00 2024-09-10T18:53:40.437 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:53:40.437 INFO:teuthology.orchestra.run.smithi119.stdout:Total 15 MB/s | 58 MB 00:03 2024-09-10T18:53:40.502 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:53:40.537 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:53:40.537 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:53:40.684 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T18:53:40.718 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T18:53:41.025 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:53:41.025 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:53:41.267 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T18:53:41.304 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 22/26 2024-09-10T18:53:41.401 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 22/26 2024-09-10T18:53:41.401 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T18:53:41.401 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T18:53:41.401 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:41.438 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T18:53:42.226 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:53:42.576 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-10T18:53:42.676 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T18:53:42.772 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T18:53:42.854 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 3/26 2024-09-10T18:53:42.902 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T18:53:42.982 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T18:53:43.028 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-10T18:53:43.065 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-10T18:53:43.506 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-10T18:53:43.530 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 8/26 2024-09-10T18:53:43.593 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 8/26 2024-09-10T18:53:43.630 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-10T18:53:43.681 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9. 10/26 2024-09-10T18:53:43.726 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 11/26 2024-09-10T18:53:43.788 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 12/26 2024-09-10T18:53:43.816 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 13/26 2024-09-10T18:53:43.882 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 13/26 2024-09-10T18:53:43.922 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-10T18:53:43.958 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-09-10T18:53:43.998 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-10T18:53:44.301 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-10T18:53:44.337 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 18/26 2024-09-10T18:53:44.427 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 18/26 2024-09-10T18:53:44.501 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 19/26 2024-09-10T18:53:44.531 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x8 20/26 2024-09-10T18:53:47.229 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T18:53:47.261 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T18:53:47.790 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T18:53:47.831 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 22/26 2024-09-10T18:53:47.935 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 22/26 2024-09-10T18:53:47.935 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T18:53:47.935 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T18:53:47.935 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:53:47.973 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T18:53:51.397 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T18:53:51.397 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-09-10T18:53:51.397 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-09-10T18:53:51.398 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-09-10T18:53:51.398 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-09-10T18:53:51.398 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-09-10T18:53:51.398 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /root 2024-09-10T18:53:51.398 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-09-10T18:53:51.398 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:52.696 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 24/26 2024-09-10T18:53:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 24/26 2024-09-10T18:53:52.778 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:53:52.778 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T18:53:52.778 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T18:53:52.778 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T18:53:52.778 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:52.780 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-10T18:53:52.871 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-10T18:53:52.871 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T18:53:54.112 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T18:53:54.112 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 1/26 2024-09-10T18:53:54.112 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T18:53:54.112 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 3/26 2024-09-10T18:53:54.112 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T18:53:54.112 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 5/26 2024-09-10T18:53:54.112 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 6/26 2024-09-10T18:53:54.112 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 7/26 2024-09-10T18:53:54.112 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9. 8/26 2024-09-10T18:53:54.112 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x8 9/26 2024-09-10T18:53:54.113 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 10/26 2024-09-10T18:53:54.113 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 11/26 2024-09-10T18:53:54.113 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 12/26 2024-09-10T18:53:54.113 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 13/26 2024-09-10T18:53:54.113 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-10T18:53:54.113 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-10T18:53:54.113 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-10T18:53:54.113 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-10T18:53:54.113 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-10T18:53:54.113 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-09-10T18:53:54.113 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-10T18:53:54.113 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-10T18:53:54.114 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-10T18:53:54.114 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T18:53:54.114 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-10T18:53:54.114 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 25/26 2024-09-10T18:53:54.933 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T18:53:54.933 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:54.934 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-09-10T18:53:54.934 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.934 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.934 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:53:54.934 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.934 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.934 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.934 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.934 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T18:53:54.934 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.934 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T18:53:54.934 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T18:53:54.934 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T18:53:54.935 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.935 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T18:53:54.935 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.935 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T18:53:54.935 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T18:53:54.935 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.935 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.935 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.935 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T18:53:54.935 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.935 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.935 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:53:54.935 INFO:teuthology.orchestra.run.smithi194.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T18:53:54.936 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:54.936 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:53:55.307 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-test 2024-09-10T18:53:55.886 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:27 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:53:56.023 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:53:56.024 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:53:56.024 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-10T18:53:56.024 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:53:56.024 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:53:56.024 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:17.2.7-1606.g5be57395.el9 ceph 48 M 2024-09-10T18:53:56.024 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-10T18:53:56.024 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite x86_64 2:17.2.7-1606.g5be57395.el9 ceph 163 k 2024-09-10T18:53:56.025 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T18:53:56.025 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T18:53:56.025 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:56.025 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:53:56.025 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:53:56.025 INFO:teuthology.orchestra.run.smithi194.stdout:Install 4 Packages 2024-09-10T18:53:56.025 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:53:56.025 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 48 M 2024-09-10T18:53:56.026 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 193 M 2024-09-10T18:53:56.026 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:53:56.343 INFO:teuthology.orchestra.run.smithi194.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.4 MB/s | 304 kB 00:00 2024-09-10T18:53:56.368 INFO:teuthology.orchestra.run.smithi194.stdout:(2/4): libcephsqlite-17.2.7-1606.g5be57395.el9. 690 kB/s | 163 kB 00:00 2024-09-10T18:53:56.485 INFO:teuthology.orchestra.run.smithi194.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 449 kB/s | 64 kB 00:00 2024-09-10T18:53:57.611 INFO:teuthology.orchestra.run.smithi194.stdout:(4/4): ceph-test-17.2.7-1606.g5be57395.el9.x86_ 32 MB/s | 48 MB 00:01 2024-09-10T18:53:57.613 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:53:57.613 INFO:teuthology.orchestra.run.smithi194.stdout:Total 31 MB/s | 48 MB 00:01 2024-09-10T18:53:57.641 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:53:57.660 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:53:57.661 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:53:57.914 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T18:53:57.914 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2024-09-10T18:53:57.914 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2024-09-10T18:53:57.914 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2024-09-10T18:53:57.914 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2024-09-10T18:53:57.914 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2024-09-10T18:53:57.914 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /root 2024-09-10T18:53:57.914 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2024-09-10T18:53:57.914 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:53:58.031 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:53:58.031 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:53:58.548 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:53:58.643 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T18:53:58.687 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T18:53:58.724 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 3/4 2024-09-10T18:53:59.248 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 24/26 2024-09-10T18:53:59.327 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 24/26 2024-09-10T18:53:59.327 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:53:59.327 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T18:53:59.327 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T18:53:59.327 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T18:53:59.327 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:53:59.329 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-10T18:53:59.418 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-10T18:53:59.418 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T18:54:00.699 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T18:54:00.700 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 1/26 2024-09-10T18:54:00.700 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T18:54:00.700 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 3/26 2024-09-10T18:54:00.700 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T18:54:00.700 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 5/26 2024-09-10T18:54:00.700 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 6/26 2024-09-10T18:54:00.700 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 7/26 2024-09-10T18:54:00.700 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9. 8/26 2024-09-10T18:54:00.700 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x8 9/26 2024-09-10T18:54:00.700 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 10/26 2024-09-10T18:54:00.701 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 11/26 2024-09-10T18:54:00.701 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 12/26 2024-09-10T18:54:00.701 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 13/26 2024-09-10T18:54:00.701 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-10T18:54:00.701 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-10T18:54:00.701 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-10T18:54:00.701 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-10T18:54:00.701 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-10T18:54:00.701 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-09-10T18:54:00.702 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-10T18:54:00.702 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-10T18:54:00.702 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-10T18:54:00.702 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T18:54:00.702 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-10T18:54:00.702 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 25/26 2024-09-10T18:54:01.597 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T18:54:01.597 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:01.597 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2024-09-10T18:54:01.598 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.598 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.598 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:54:01.598 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.598 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.598 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.598 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.598 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T18:54:01.598 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.598 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T18:54:01.598 INFO:teuthology.orchestra.run.smithi119.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T18:54:01.599 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T18:54:01.599 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.599 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T18:54:01.599 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.599 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T18:54:01.599 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T18:54:01.599 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.599 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.599 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.599 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T18:54:01.600 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.600 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.600 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:01.600 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T18:54:01.600 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:01.600 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:54:01.951 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-test 2024-09-10T18:54:02.528 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:28 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:54:02.672 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:54:02.673 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:02.673 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-10T18:54:02.673 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:02.673 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:54:02.673 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:17.2.7-1606.g5be57395.el9 ceph 48 M 2024-09-10T18:54:02.673 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-10T18:54:02.673 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:17.2.7-1606.g5be57395.el9 ceph 163 k 2024-09-10T18:54:02.673 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T18:54:02.673 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T18:54:02.674 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:02.674 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:54:02.674 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:02.674 INFO:teuthology.orchestra.run.smithi119.stdout:Install 4 Packages 2024-09-10T18:54:02.674 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:02.674 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 48 M 2024-09-10T18:54:02.674 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 193 M 2024-09-10T18:54:02.675 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:54:02.938 INFO:teuthology.orchestra.run.smithi119.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.1 MB/s | 304 kB 00:00 2024-09-10T18:54:03.013 INFO:teuthology.orchestra.run.smithi119.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 852 kB/s | 64 kB 00:00 2024-09-10T18:54:03.038 INFO:teuthology.orchestra.run.smithi119.stdout:(3/4): libcephsqlite-17.2.7-1606.g5be57395.el9. 673 kB/s | 163 kB 00:00 2024-09-10T18:54:03.119 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 3/4 2024-09-10T18:54:03.197 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 4/4 2024-09-10T18:54:04.289 INFO:teuthology.orchestra.run.smithi119.stdout:(4/4): ceph-test-17.2.7-1606.g5be57395.el9.x86_ 32 MB/s | 48 MB 00:01 2024-09-10T18:54:04.291 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:04.291 INFO:teuthology.orchestra.run.smithi119.stdout:Total 30 MB/s | 48 MB 00:01 2024-09-10T18:54:04.319 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:54:04.338 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:54:04.338 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:54:04.703 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:54:04.704 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:54:05.219 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:54:05.291 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T18:54:05.329 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T18:54:05.357 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 3/4 2024-09-10T18:54:05.422 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 4/4 2024-09-10T18:54:05.422 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 1/4 2024-09-10T18:54:05.422 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 2/4 2024-09-10T18:54:05.423 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T18:54:05.762 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T18:54:05.763 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:05.763 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:54:05.763 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:05.763 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:05.763 INFO:teuthology.orchestra.run.smithi194.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T18:54:05.763 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T18:54:05.763 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:05.764 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:54:06.031 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph 2024-09-10T18:54:06.605 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:38 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:54:06.750 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:54:06.752 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:06.752 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-10T18:54:06.752 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:06.752 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:54:06.752 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:17.2.7-1606.g5be57395.el9 ceph 6.5 k 2024-09-10T18:54:06.752 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-10T18:54:06.753 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.1 M 2024-09-10T18:54:06.753 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:17.2.7-1606.g5be57395.el9 ceph 1.4 M 2024-09-10T18:54:06.753 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 240 k 2024-09-10T18:54:06.753 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:17.2.7-1606.g5be57395.el9 ceph 3.9 M 2024-09-10T18:54:06.753 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 15 M 2024-09-10T18:54:06.753 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T18:54:06.753 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T18:54:06.753 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T18:54:06.753 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T18:54:06.753 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T18:54:06.753 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T18:54:06.754 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T18:54:06.754 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections 2024-09-10T18:54:06.754 INFO:teuthology.orchestra.run.smithi194.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-10T18:54:06.754 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T18:54:06.754 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T18:54:06.754 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T18:54:06.754 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T18:54:06.754 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T18:54:06.754 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T18:54:06.755 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T18:54:06.755 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T18:54:06.755 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T18:54:06.755 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T18:54:06.755 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T18:54:06.755 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T18:54:06.755 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T18:54:06.755 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T18:54:06.755 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T18:54:06.755 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T18:54:06.755 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T18:54:06.756 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T18:54:06.756 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T18:54:06.756 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T18:54:06.756 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:06.756 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:54:06.756 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:06.756 INFO:teuthology.orchestra.run.smithi194.stdout:Install 34 Packages 2024-09-10T18:54:06.756 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:06.757 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 27 M 2024-09-10T18:54:06.757 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 96 M 2024-09-10T18:54:06.757 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:54:06.989 INFO:teuthology.orchestra.run.smithi194.stdout:(1/34): ceph-17.2.7-1606.g5be57395.el9.x86_64.r 50 kB/s | 6.5 kB 00:00 2024-09-10T18:54:07.181 INFO:teuthology.orchestra.run.smithi194.stdout:(2/34): ceph-mgr-17.2.7-1606.g5be57395.el9.x86_ 4.4 MB/s | 1.4 MB 00:00 2024-09-10T18:54:07.223 INFO:teuthology.orchestra.run.smithi194.stdout:(3/34): ceph-mds-17.2.7-1606.g5be57395.el9.x86_ 5.7 MB/s | 2.1 MB 00:00 2024-09-10T18:54:07.313 INFO:teuthology.orchestra.run.smithi194.stdout:(4/34): ceph-mon-17.2.7-1606.g5be57395.el9.x86_ 12 MB/s | 3.9 MB 00:00 2024-09-10T18:54:07.338 INFO:teuthology.orchestra.run.smithi194.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1606.g5be5 2.0 MB/s | 240 kB 00:00 2024-09-10T18:54:07.472 INFO:teuthology.orchestra.run.smithi194.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.6 MB/s | 253 kB 00:00 2024-09-10T18:54:07.514 INFO:teuthology.orchestra.run.smithi194.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-10T18:54:07.556 INFO:teuthology.orchestra.run.smithi194.stdout:(8/34): python3-cryptography-36.0.1-4.el9.x86_6 5.6 MB/s | 1.2 MB 00:00 2024-09-10T18:54:07.581 INFO:teuthology.orchestra.run.smithi194.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 2.0 MB/s | 135 kB 00:00 2024-09-10T18:54:07.607 INFO:teuthology.orchestra.run.smithi194.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 2.4 MB/s | 125 kB 00:00 2024-09-10T18:54:07.632 INFO:teuthology.orchestra.run.smithi194.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-10T18:54:07.782 INFO:teuthology.orchestra.run.smithi194.stdout:(12/34): ceph-osd-17.2.7-1606.g5be57395.el9.x86 26 MB/s | 15 MB 00:00 2024-09-10T18:54:07.808 INFO:teuthology.orchestra.run.smithi194.stdout:(13/34): python3-mako-1.1.4-6.el9.noarch.rpm 857 kB/s | 172 kB 00:00 2024-09-10T18:54:07.824 INFO:teuthology.orchestra.run.smithi194.stdout:(14/34): python3-markupsafe-1.1.1-12.el9.x86_64 180 kB/s | 35 kB 00:00 2024-09-10T18:54:07.842 INFO:teuthology.orchestra.run.smithi194.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 1.5 MB/s | 51 kB 00:00 2024-09-10T18:54:07.858 INFO:teuthology.orchestra.run.smithi194.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-09-10T18:54:07.900 INFO:teuthology.orchestra.run.smithi194.stdout:(17/34): python3-packaging-20.9-5.el9.noarch.rp 658 kB/s | 77 kB 00:00 2024-09-10T18:54:07.917 INFO:teuthology.orchestra.run.smithi194.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 577 kB/s | 43 kB 00:00 2024-09-10T18:54:07.942 INFO:teuthology.orchestra.run.smithi194.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 592 kB/s | 11 kB 00:00 2024-09-10T18:54:07.968 INFO:teuthology.orchestra.run.smithi194.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-10T18:54:07.993 INFO:teuthology.orchestra.run.smithi194.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 363 kB/s | 18 kB 00:00 2024-09-10T18:54:08.018 INFO:teuthology.orchestra.run.smithi194.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-10T18:54:08.043 INFO:teuthology.orchestra.run.smithi194.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-10T18:54:08.069 INFO:teuthology.orchestra.run.smithi194.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-09-10T18:54:08.094 INFO:teuthology.orchestra.run.smithi194.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 924 kB/s | 46 kB 00:00 2024-09-10T18:54:08.119 INFO:teuthology.orchestra.run.smithi194.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-10T18:54:08.144 INFO:teuthology.orchestra.run.smithi194.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 660 kB/s | 16 kB 00:00 2024-09-10T18:54:08.178 INFO:teuthology.orchestra.run.smithi194.stdout:(28/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-10T18:54:08.203 INFO:teuthology.orchestra.run.smithi194.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-10T18:54:08.229 INFO:teuthology.orchestra.run.smithi194.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-09-10T18:54:08.262 INFO:teuthology.orchestra.run.smithi194.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-09-10T18:54:08.287 INFO:teuthology.orchestra.run.smithi194.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 800 kB/s | 20 kB 00:00 2024-09-10T18:54:08.313 INFO:teuthology.orchestra.run.smithi194.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-10T18:54:08.338 INFO:teuthology.orchestra.run.smithi194.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-10T18:54:08.342 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:08.342 INFO:teuthology.orchestra.run.smithi194.stdout:Total 17 MB/s | 27 MB 00:01 2024-09-10T18:54:08.467 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:54:08.504 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:54:08.504 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:54:08.882 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:54:08.882 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:54:09.768 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 3/4 2024-09-10T18:54:09.849 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 4/4 2024-09-10T18:54:10.129 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:54:10.182 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-10T18:54:10.214 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-10T18:54:10.248 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-10T18:54:10.340 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-10T18:54:10.417 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-10T18:54:10.467 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-10T18:54:10.510 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-10T18:54:10.540 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-10T18:54:10.574 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-10T18:54:10.608 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-10T18:54:10.648 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-10T18:54:10.700 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-10T18:54:10.733 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-10T18:54:10.782 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-10T18:54:10.829 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-10T18:54:10.912 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-10T18:54:11.059 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-10T18:54:11.147 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-10T18:54:11.208 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-10T18:54:11.247 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-10T18:54:11.296 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-10T18:54:11.351 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-10T18:54:11.546 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-10T18:54:11.600 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-10T18:54:11.672 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-10T18:54:11.759 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-10T18:54:11.805 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-10T18:54:11.924 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-10T18:54:12.071 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9. 29/34 2024-09-10T18:54:12.103 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 30/34 2024-09-10T18:54:12.129 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 30/34 2024-09-10T18:54:12.129 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:54:12.129 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T18:54:12.130 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T18:54:12.130 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T18:54:12.130 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 4/4 2024-09-10T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 1/4 2024-09-10T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 2/4 2024-09-10T18:54:12.154 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T18:54:12.500 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:12.501 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:54:12.784 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph 2024-09-10T18:54:13.357 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:39 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:54:13.491 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:54:13.493 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:13.493 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-10T18:54:13.494 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:13.494 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:54:13.494 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:17.2.7-1606.g5be57395.el9 ceph 6.5 k 2024-09-10T18:54:13.494 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-10T18:54:13.494 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.1 M 2024-09-10T18:54:13.494 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:17.2.7-1606.g5be57395.el9 ceph 1.4 M 2024-09-10T18:54:13.494 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 240 k 2024-09-10T18:54:13.494 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:17.2.7-1606.g5be57395.el9 ceph 3.9 M 2024-09-10T18:54:13.494 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 15 M 2024-09-10T18:54:13.494 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T18:54:13.494 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections 2024-09-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi119.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T18:54:13.495 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T18:54:13.496 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T18:54:13.497 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T18:54:13.497 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T18:54:13.497 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T18:54:13.497 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T18:54:13.497 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:13.497 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:54:13.497 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:13.497 INFO:teuthology.orchestra.run.smithi119.stdout:Install 34 Packages 2024-09-10T18:54:13.497 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:13.498 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 27 M 2024-09-10T18:54:13.498 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 96 M 2024-09-10T18:54:13.498 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:54:13.504 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 31/34 2024-09-10T18:54:13.534 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 31/34 2024-09-10T18:54:13.534 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:54:13.534 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T18:54:13.535 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T18:54:13.535 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T18:54:13.535 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:13.913 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 32/34 2024-09-10T18:54:13.938 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 32/34 2024-09-10T18:54:13.938 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:54:13.938 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T18:54:13.938 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T18:54:13.938 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T18:54:13.939 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:14.084 INFO:teuthology.orchestra.run.smithi119.stdout:(1/34): ceph-17.2.7-1606.g5be57395.el9.x86_64.r 46 kB/s | 6.5 kB 00:00 2024-09-10T18:54:14.139 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 33/34 2024-09-10T18:54:14.162 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 33/34 2024-09-10T18:54:14.162 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:54:14.163 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T18:54:14.163 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T18:54:14.163 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T18:54:14.163 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:14.242 INFO:teuthology.orchestra.run.smithi119.stdout:(2/34): ceph-mgr-17.2.7-1606.g5be57395.el9.x86_ 4.7 MB/s | 1.4 MB 00:00 2024-09-10T18:54:14.245 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:17.2.7-1606.g5be57395.el9.x86_64 34/34 2024-09-10T18:54:14.292 INFO:teuthology.orchestra.run.smithi119.stdout:(3/34): ceph-mds-17.2.7-1606.g5be57395.el9.x86_ 5.9 MB/s | 2.1 MB 00:00 2024-09-10T18:54:14.418 INFO:teuthology.orchestra.run.smithi119.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1606.g5be5 1.9 MB/s | 240 kB 00:00 2024-09-10T18:54:14.543 INFO:teuthology.orchestra.run.smithi119.stdout:(5/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-09-10T18:54:14.760 INFO:teuthology.orchestra.run.smithi119.stdout:(6/34): ceph-osd-17.2.7-1606.g5be57395.el9.x86_ 30 MB/s | 15 MB 00:00 2024-09-10T18:54:14.818 INFO:teuthology.orchestra.run.smithi119.stdout:(7/34): ceph-mon-17.2.7-1606.g5be57395.el9.x86_ 5.3 MB/s | 3.9 MB 00:00 2024-09-10T18:54:14.885 INFO:teuthology.orchestra.run.smithi119.stdout:(8/34): python3-cryptography-36.0.1-4.el9.x86_6 3.6 MB/s | 1.2 MB 00:00 2024-09-10T18:54:14.944 INFO:teuthology.orchestra.run.smithi119.stdout:(9/34): python3-ply-3.11-14.el9.noarch.rpm 580 kB/s | 106 kB 00:00 2024-09-10T18:54:14.986 INFO:teuthology.orchestra.run.smithi119.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 1.2 MB/s | 125 kB 00:00 2024-09-10T18:54:15.019 INFO:teuthology.orchestra.run.smithi119.stdout:(11/34): python3-pycparser-2.20-6.el9.noarch.rp 673 kB/s | 135 kB 00:00 2024-09-10T18:54:15.045 INFO:teuthology.orchestra.run.smithi119.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 2.1 MB/s | 215 kB 00:00 2024-09-10T18:54:15.120 INFO:teuthology.orchestra.run.smithi119.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 347 kB/s | 35 kB 00:00 2024-09-10T18:54:15.173 INFO:teuthology.orchestra.run.smithi119.stdout:(14/34): python3-packaging-20.9-5.el9.noarch.rp 617 kB/s | 77 kB 00:00 2024-09-10T18:54:15.195 INFO:teuthology.orchestra.run.smithi119.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 677 kB/s | 51 kB 00:00 2024-09-10T18:54:15.221 INFO:teuthology.orchestra.run.smithi119.stdout:(16/34): python3-mako-1.1.4-6.el9.noarch.rpm 734 kB/s | 172 kB 00:00 2024-09-10T18:54:15.246 INFO:teuthology.orchestra.run.smithi119.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-10T18:54:15.288 INFO:teuthology.orchestra.run.smithi119.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 471 kB/s | 43 kB 00:00 2024-09-10T18:54:15.305 INFO:teuthology.orchestra.run.smithi119.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 712 kB/s | 11 kB 00:00 2024-09-10T18:54:15.330 INFO:teuthology.orchestra.run.smithi119.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-10T18:54:15.347 INFO:teuthology.orchestra.run.smithi119.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 424 kB/s | 18 kB 00:00 2024-09-10T18:54:15.364 INFO:teuthology.orchestra.run.smithi119.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 690 kB/s | 23 kB 00:00 2024-09-10T18:54:15.381 INFO:teuthology.orchestra.run.smithi119.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 578 kB/s | 19 kB 00:00 2024-09-10T18:54:15.398 INFO:teuthology.orchestra.run.smithi119.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 587 kB/s | 20 kB 00:00 2024-09-10T18:54:15.425 INFO:teuthology.orchestra.run.smithi119.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 1.0 MB/s | 46 kB 00:00 2024-09-10T18:54:15.451 INFO:teuthology.orchestra.run.smithi119.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-10T18:54:15.476 INFO:teuthology.orchestra.run.smithi119.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-09-10T18:54:15.501 INFO:teuthology.orchestra.run.smithi119.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-10T18:54:15.526 INFO:teuthology.orchestra.run.smithi119.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-10T18:54:15.552 INFO:teuthology.orchestra.run.smithi119.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 709 kB/s | 36 kB 00:00 2024-09-10T18:54:15.577 INFO:teuthology.orchestra.run.smithi119.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-10T18:54:15.602 INFO:teuthology.orchestra.run.smithi119.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-09-10T18:54:15.628 INFO:teuthology.orchestra.run.smithi119.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-10T18:54:15.661 INFO:teuthology.orchestra.run.smithi119.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-10T18:54:15.666 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:15.666 INFO:teuthology.orchestra.run.smithi119.stdout:Total 13 MB/s | 27 MB 00:02 2024-09-10T18:54:15.795 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:54:15.832 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:54:15.832 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:54:16.002 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-2:17.2.7-1606.g5be57395.el9.x86_64 34/34 2024-09-10T18:54:16.002 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:17.2.7-1606.g5be57395.el9.x86_64 1/34 2024-09-10T18:54:16.003 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 2/34 2024-09-10T18:54:16.003 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 3/34 2024-09-10T18:54:16.003 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 4/34 2024-09-10T18:54:16.003 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 5/34 2024-09-10T18:54:16.003 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9. 6/34 2024-09-10T18:54:16.003 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-10T18:54:16.003 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-10T18:54:16.003 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-10T18:54:16.003 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-10T18:54:16.003 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-10T18:54:16.003 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-10T18:54:16.003 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-10T18:54:16.004 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-10T18:54:16.004 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-10T18:54:16.004 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-10T18:54:16.004 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-10T18:54:16.004 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-10T18:54:16.004 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-10T18:54:16.004 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-10T18:54:16.004 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-10T18:54:16.004 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-10T18:54:16.004 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-10T18:54:16.004 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-10T18:54:16.004 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-10T18:54:16.004 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-10T18:54:16.005 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-10T18:54:16.005 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-10T18:54:16.005 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-10T18:54:16.005 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-10T18:54:16.005 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-10T18:54:16.005 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-10T18:54:16.005 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-10T18:54:16.208 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:54:16.208 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:54:17.082 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-10T18:54:17.082 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:17.082 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:54:17.082 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:17.082 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:17.082 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:17.082 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:17.082 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:17.083 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:17.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T18:54:17.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T18:54:17.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T18:54:17.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T18:54:17.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T18:54:17.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T18:54:17.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T18:54:17.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T18:54:17.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T18:54:17.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T18:54:17.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T18:54:17.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T18:54:17.084 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T18:54:17.084 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T18:54:17.084 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T18:54:17.084 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T18:54:17.084 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T18:54:17.084 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T18:54:17.084 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T18:54:17.084 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T18:54:17.084 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T18:54:17.084 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T18:54:17.084 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T18:54:17.084 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T18:54:17.084 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T18:54:17.085 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T18:54:17.085 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T18:54:17.085 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T18:54:17.085 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:17.085 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:54:17.443 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:54:17.468 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-base 2024-09-10T18:54:17.506 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-10T18:54:17.546 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-10T18:54:17.579 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-10T18:54:17.671 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-10T18:54:17.750 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-10T18:54:17.799 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-10T18:54:17.842 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-10T18:54:17.872 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-10T18:54:17.906 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-10T18:54:17.939 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-10T18:54:17.980 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-10T18:54:18.025 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-10T18:54:18.038 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:50 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:54:18.083 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-10T18:54:18.129 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:54:18.132 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-10T18:54:18.170 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:54:18.171 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-10T18:54:18.171 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:54:18.173 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-10T18:54:18.237 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install cephadm 2024-09-10T18:54:18.253 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-10T18:54:18.401 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-10T18:54:18.488 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-10T18:54:18.541 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-10T18:54:18.579 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-10T18:54:18.628 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-10T18:54:18.684 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-10T18:54:18.803 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:50 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:54:18.880 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-10T18:54:18.925 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-10T18:54:18.934 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:54:18.935 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:18.936 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-10T18:54:18.936 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:18.936 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:54:18.936 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 83 k 2024-09-10T18:54:18.936 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:18.936 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:54:18.936 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:18.937 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-10T18:54:18.937 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:18.937 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 83 k 2024-09-10T18:54:18.937 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 375 k 2024-09-10T18:54:18.937 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:54:19.004 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-10T18:54:19.092 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-10T18:54:19.128 INFO:teuthology.orchestra.run.smithi194.stdout:cephadm-17.2.7-1606.g5be57395.el9.noarch.rpm 429 kB/s | 83 kB 00:00 2024-09-10T18:54:19.129 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:19.129 INFO:teuthology.orchestra.run.smithi194.stdout:Total 426 kB/s | 83 kB 00:00 2024-09-10T18:54:19.129 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:54:19.134 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:54:19.135 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:54:19.137 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:54:19.137 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:54:19.142 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-10T18:54:19.217 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:54:19.256 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-10T18:54:19.413 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9. 29/34 2024-09-10T18:54:19.451 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 30/34 2024-09-10T18:54:19.476 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 30/34 2024-09-10T18:54:19.477 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:54:19.477 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T18:54:19.477 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T18:54:19.477 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T18:54:19.477 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:19.636 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:54:19.723 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:54:20.099 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:54:20.343 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:54:20.343 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:20.343 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:54:20.343 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:20.343 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:20.343 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:54:20.540 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T18:54:20.828 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 31/34 2024-09-10T18:54:20.857 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 31/34 2024-09-10T18:54:20.858 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:54:20.858 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T18:54:20.858 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T18:54:20.858 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T18:54:20.858 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:21.116 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:53 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:54:21.250 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:54:21.250 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:21.251 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-09-10T18:54:21.251 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:21.251 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:54:21.251 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1606.g5be57395.el9 ceph 137 k 2024-09-10T18:54:21.251 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:21.251 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:54:21.251 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:21.251 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-10T18:54:21.251 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:21.251 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 137 k 2024-09-10T18:54:21.251 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 379 k 2024-09-10T18:54:21.251 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:54:21.253 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 32/34 2024-09-10T18:54:21.278 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 32/34 2024-09-10T18:54:21.278 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:54:21.278 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T18:54:21.278 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T18:54:21.278 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T18:54:21.279 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:21.443 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-immutable-object-cache-17.2.7-1606.g5be573 718 kB/s | 137 kB 00:00 2024-09-10T18:54:21.444 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:21.444 INFO:teuthology.orchestra.run.smithi194.stdout:Total 712 kB/s | 137 kB 00:00 2024-09-10T18:54:21.444 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:54:21.451 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:54:21.452 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:54:21.479 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 33/34 2024-09-10T18:54:21.503 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 33/34 2024-09-10T18:54:21.503 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:54:21.503 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T18:54:21.503 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T18:54:21.503 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T18:54:21.503 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:21.513 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:54:21.513 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:54:21.568 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:17.2.7-1606.g5be57395.el9.x86_64 34/34 2024-09-10T18:54:21.664 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:54:21.735 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T18:54:21.760 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T18:54:21.769 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:54:21.769 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T18:54:21.769 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:22.732 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T18:54:22.732 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:22.733 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:54:22.733 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:22.733 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:22.733 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:54:22.953 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr 2024-09-10T18:54:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-2:17.2.7-1606.g5be57395.el9.x86_64 34/34 2024-09-10T18:54:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:17.2.7-1606.g5be57395.el9.x86_64 1/34 2024-09-10T18:54:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 2/34 2024-09-10T18:54:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 3/34 2024-09-10T18:54:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 4/34 2024-09-10T18:54:23.335 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 5/34 2024-09-10T18:54:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9. 6/34 2024-09-10T18:54:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-10T18:54:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-10T18:54:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-10T18:54:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-10T18:54:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-10T18:54:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-10T18:54:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-10T18:54:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-10T18:54:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-10T18:54:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-10T18:54:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-10T18:54:23.336 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-10T18:54:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-10T18:54:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-10T18:54:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-10T18:54:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-10T18:54:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-10T18:54:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-10T18:54:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-10T18:54:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-10T18:54:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-10T18:54:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-10T18:54:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-10T18:54:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-10T18:54:23.337 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-10T18:54:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-10T18:54:23.338 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-10T18:54:23.536 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:55 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:54:23.632 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:54:23.674 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:54:23.675 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-10T18:54:23.675 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:54:23.739 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T18:54:24.248 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-10T18:54:24.248 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:24.248 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:54:24.248 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:24.248 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:24.248 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:24.249 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:24.249 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:24.249 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:24.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T18:54:24.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T18:54:24.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T18:54:24.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T18:54:24.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T18:54:24.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T18:54:24.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T18:54:24.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T18:54:24.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T18:54:24.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T18:54:24.249 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T18:54:24.250 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T18:54:24.250 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T18:54:24.250 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T18:54:24.250 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T18:54:24.250 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T18:54:24.250 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T18:54:24.250 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T18:54:24.250 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T18:54:24.250 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T18:54:24.250 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T18:54:24.250 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T18:54:24.250 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T18:54:24.250 INFO:teuthology.orchestra.run.smithi119.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T18:54:24.251 INFO:teuthology.orchestra.run.smithi119.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T18:54:24.251 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T18:54:24.251 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T18:54:24.251 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T18:54:24.251 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:24.251 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:54:24.299 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:56 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:54:24.432 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:54:24.432 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:24.432 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-10T18:54:24.433 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:24.433 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:54:24.433 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 3.4 M 2024-09-10T18:54:24.433 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-10T18:54:24.433 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 24 k 2024-09-10T18:54:24.433 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 15 k 2024-09-10T18:54:24.433 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-10T18:54:24.433 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T18:54:24.433 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T18:54:24.433 INFO:teuthology.orchestra.run.smithi194.stdout:Installing weak dependencies: 2024-09-10T18:54:24.433 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-10T18:54:24.433 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:24.433 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:54:24.434 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:24.434 INFO:teuthology.orchestra.run.smithi194.stdout:Install 7 Packages 2024-09-10T18:54:24.434 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:24.434 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 3.7 M 2024-09-10T18:54:24.434 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 80 M 2024-09-10T18:54:24.434 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:54:24.585 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-base 2024-09-10T18:54:24.996 INFO:teuthology.orchestra.run.smithi194.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1606.g5be5 93 kB/s | 15 kB 00:00 2024-09-10T18:54:25.022 INFO:teuthology.orchestra.run.smithi194.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1606.g5be 133 kB/s | 24 kB 00:00 2024-09-10T18:54:25.063 INFO:teuthology.orchestra.run.smithi194.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-10T18:54:25.089 INFO:teuthology.orchestra.run.smithi194.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-10T18:54:25.114 INFO:teuthology.orchestra.run.smithi194.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-09-10T18:54:25.147 INFO:teuthology.orchestra.run.smithi194.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.2 MB/s | 188 kB 00:00 2024-09-10T18:54:25.160 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:51 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:54:25.231 INFO:teuthology.orchestra.run.smithi194.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1606.g5be57395 8.5 MB/s | 3.4 MB 00:00 2024-09-10T18:54:25.234 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:25.235 INFO:teuthology.orchestra.run.smithi194.stdout:Total 4.6 MB/s | 3.7 MB 00:00 2024-09-10T18:54:25.235 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:54:25.243 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:54:25.243 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:54:25.250 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:54:25.292 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:54:25.293 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-10T18:54:25.293 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:54:25.339 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:54:25.339 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:54:25.357 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install cephadm 2024-09-10T18:54:25.644 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:54:25.716 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-10T18:54:25.749 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-10T18:54:25.796 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-10T18:54:25.836 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-10T18:54:25.865 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.n 5/7 2024-09-10T18:54:25.916 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:51 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:54:26.050 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:54:26.051 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:26.051 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-10T18:54:26.051 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:26.052 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:54:26.052 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 83 k 2024-09-10T18:54:26.052 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:26.052 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:54:26.052 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:26.052 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-10T18:54:26.052 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:26.052 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 83 k 2024-09-10T18:54:26.052 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 375 k 2024-09-10T18:54:26.052 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:54:26.237 INFO:teuthology.orchestra.run.smithi119.stdout:cephadm-17.2.7-1606.g5be57395.el9.noarch.rpm 449 kB/s | 83 kB 00:00 2024-09-10T18:54:26.237 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:26.237 INFO:teuthology.orchestra.run.smithi119.stdout:Total 445 kB/s | 83 kB 00:00 2024-09-10T18:54:26.238 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:54:26.243 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:54:26.243 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:54:26.245 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:54:26.245 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:54:26.324 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:54:26.777 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:54:26.851 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9. 6/7 2024-09-10T18:54:26.873 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:54:26.894 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 7/7 2024-09-10T18:54:27.242 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:54:27.471 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:54:27.471 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:27.471 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:54:27.471 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:27.471 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:27.472 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:54:27.690 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T18:54:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 7/7 2024-09-10T18:54:27.886 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9. 1/7 2024-09-10T18:54:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 2/7 2024-09-10T18:54:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.n 3/7 2024-09-10T18:54:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-10T18:54:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-10T18:54:27.887 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-10T18:54:28.168 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-10T18:54:28.168 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:28.168 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:54:28.168 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:28.168 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:28.168 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:28.169 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-10T18:54:28.169 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-10T18:54:28.169 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T18:54:28.169 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T18:54:28.169 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:28.169 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:54:28.274 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:54 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:54:28.409 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:54:28.410 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:28.410 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-09-10T18:54:28.410 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:28.410 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:54:28.410 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1606.g5be57395.el9 ceph 137 k 2024-09-10T18:54:28.410 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:28.410 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:54:28.410 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:28.410 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-10T18:54:28.410 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:28.411 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 137 k 2024-09-10T18:54:28.411 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 379 k 2024-09-10T18:54:28.411 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:54:28.411 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T18:54:28.617 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-immutable-object-cache-17.2.7-1606.g5be573 666 kB/s | 137 kB 00:00 2024-09-10T18:54:28.618 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:28.618 INFO:teuthology.orchestra.run.smithi119.stdout:Total 661 kB/s | 137 kB 00:00 2024-09-10T18:54:28.618 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:54:28.626 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:54:28.626 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:54:28.687 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:54:28.688 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:54:28.851 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:54:28.944 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T18:54:28.969 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T18:54:28.969 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:54:28.969 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T18:54:28.969 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:28.975 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:00 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:54:29.109 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:54:29.110 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-09-10T18:54:29.110 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-10T18:54:29.111 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-09-10T18:54:29.111 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:54:29.111 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 7.4 M 2024-09-10T18:54:29.111 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-10T18:54:29.111 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T18:54:29.111 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T18:54:29.111 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T18:54:29.111 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T18:54:29.111 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T18:54:29.111 INFO:teuthology.orchestra.run.smithi194.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T18:54:29.111 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T18:54:29.111 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T18:54:29.112 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T18:54:29.112 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T18:54:29.112 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T18:54:29.112 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:29.112 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:54:29.112 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-09-10T18:54:29.112 INFO:teuthology.orchestra.run.smithi194.stdout:Install 12 Packages 2024-09-10T18:54:29.112 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:29.113 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 38 M 2024-09-10T18:54:29.113 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 212 M 2024-09-10T18:54:29.113 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:54:29.472 INFO:teuthology.orchestra.run.smithi194.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-10T18:54:29.597 INFO:teuthology.orchestra.run.smithi194.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 262 kB/s | 33 kB 00:00 2024-09-10T18:54:29.631 INFO:teuthology.orchestra.run.smithi194.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.3 MB/s | 801 kB 00:00 2024-09-10T18:54:29.764 INFO:teuthology.orchestra.run.smithi194.stdout:(4/12): ceph-mgr-diskprediction-local-17.2.7-16 15 MB/s | 7.4 MB 00:00 2024-09-10T18:54:29.848 INFO:teuthology.orchestra.run.smithi194.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 84 kB/s | 18 kB 00:00 2024-09-10T18:54:29.881 INFO:teuthology.orchestra.run.smithi194.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 329 kB/s | 38 kB 00:00 2024-09-10T18:54:29.964 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T18:54:29.965 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:29.965 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:54:29.965 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:29.965 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:29.965 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:54:30.024 INFO:teuthology.orchestra.run.smithi194.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-09-10T18:54:30.186 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr 2024-09-10T18:54:30.457 INFO:teuthology.orchestra.run.smithi194.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 3.5 MB/s | 3.0 MB 00:00 2024-09-10T18:54:30.607 INFO:teuthology.orchestra.run.smithi194.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.6 MB/s | 250 kB 00:00 2024-09-10T18:54:30.771 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:56 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:54:30.863 INFO:teuthology.orchestra.run.smithi119.stdout:Package ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:54:30.905 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:54:30.906 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-10T18:54:30.906 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:54:30.972 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T18:54:31.091 INFO:teuthology.orchestra.run.smithi194.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.8 MB/s | 5.1 MB 00:01 2024-09-10T18:54:31.158 INFO:teuthology.orchestra.run.smithi194.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.8 MB/s | 4.9 MB 00:01 2024-09-10T18:54:31.561 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:57 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:54:31.694 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:54:31.695 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:31.695 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-10T18:54:31.695 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:31.696 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:54:31.696 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 3.4 M 2024-09-10T18:54:31.696 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-10T18:54:31.696 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 24 k 2024-09-10T18:54:31.696 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 15 k 2024-09-10T18:54:31.696 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-10T18:54:31.696 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T18:54:31.696 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T18:54:31.696 INFO:teuthology.orchestra.run.smithi119.stdout:Installing weak dependencies: 2024-09-10T18:54:31.696 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-10T18:54:31.696 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:31.696 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:54:31.696 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:31.697 INFO:teuthology.orchestra.run.smithi119.stdout:Install 7 Packages 2024-09-10T18:54:31.697 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:31.697 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 3.7 M 2024-09-10T18:54:31.697 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 80 M 2024-09-10T18:54:31.697 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:54:32.054 INFO:teuthology.orchestra.run.smithi119.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1606.g5be 160 kB/s | 24 kB 00:00 2024-09-10T18:54:32.079 INFO:teuthology.orchestra.run.smithi119.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1606.g5be5 82 kB/s | 15 kB 00:00 2024-09-10T18:54:32.121 INFO:teuthology.orchestra.run.smithi119.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-10T18:54:32.147 INFO:teuthology.orchestra.run.smithi119.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-10T18:54:32.172 INFO:teuthology.orchestra.run.smithi119.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-09-10T18:54:32.197 INFO:teuthology.orchestra.run.smithi119.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-10T18:54:32.322 INFO:teuthology.orchestra.run.smithi119.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1606.g5be57395 8.0 MB/s | 3.4 MB 00:00 2024-09-10T18:54:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:Total 5.8 MB/s | 3.7 MB 00:00 2024-09-10T18:54:32.326 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:54:32.334 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:54:32.334 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:54:32.435 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:54:32.436 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:54:32.500 INFO:teuthology.orchestra.run.smithi194.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.6 MB/s | 16 MB 00:01 2024-09-10T18:54:32.501 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:32.501 INFO:teuthology.orchestra.run.smithi194.stdout:Total 11 MB/s | 38 MB 00:03 2024-09-10T18:54:32.770 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:54:32.840 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-10T18:54:32.855 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:54:32.874 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-10T18:54:32.881 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:54:32.881 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:54:32.929 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-10T18:54:32.961 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-10T18:54:32.990 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.n 5/7 2024-09-10T18:54:33.397 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:54:33.397 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:54:33.989 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9. 6/7 2024-09-10T18:54:34.035 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 7/7 2024-09-10T18:54:34.163 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:54:34.264 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T18:54:34.316 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T18:54:34.414 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T18:54:34.448 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T18:54:34.634 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T18:54:34.664 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T18:54:34.749 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T18:54:35.053 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 7/7 2024-09-10T18:54:35.053 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9. 1/7 2024-09-10T18:54:35.053 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 2/7 2024-09-10T18:54:35.053 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.n 3/7 2024-09-10T18:54:35.053 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-10T18:54:35.053 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-10T18:54:35.054 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-10T18:54:35.345 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-10T18:54:35.345 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:35.345 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:54:35.345 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:35.345 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:35.346 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:35.346 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-10T18:54:35.346 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-10T18:54:35.346 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T18:54:35.346 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T18:54:35.346 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:35.346 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:54:35.400 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T18:54:35.518 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T18:54:35.559 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T18:54:36.120 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:02 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:54:36.252 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:54:36.253 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================= 2024-09-10T18:54:36.253 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-10T18:54:36.253 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================= 2024-09-10T18:54:36.253 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:54:36.253 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 7.4 M 2024-09-10T18:54:36.253 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-10T18:54:36.253 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T18:54:36.253 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T18:54:36.254 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T18:54:36.254 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T18:54:36.254 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T18:54:36.254 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T18:54:36.254 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T18:54:36.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T18:54:36.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T18:54:36.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T18:54:36.254 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T18:54:36.254 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:36.254 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:54:36.254 INFO:teuthology.orchestra.run.smithi119.stdout:======================================================================================= 2024-09-10T18:54:36.255 INFO:teuthology.orchestra.run.smithi119.stdout:Install 12 Packages 2024-09-10T18:54:36.255 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:36.255 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 38 M 2024-09-10T18:54:36.255 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 212 M 2024-09-10T18:54:36.255 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:54:36.756 INFO:teuthology.orchestra.run.smithi119.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 850 kB/s | 191 kB 00:00 2024-09-10T18:54:36.758 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T18:54:36.856 INFO:teuthology.orchestra.run.smithi119.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.4 MB/s | 801 kB 00:00 2024-09-10T18:54:37.048 INFO:teuthology.orchestra.run.smithi119.stdout:(3/12): ceph-mgr-diskprediction-local-17.2.7-16 14 MB/s | 7.4 MB 00:00 2024-09-10T18:54:37.073 INFO:teuthology.orchestra.run.smithi119.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 103 kB/s | 33 kB 00:00 2024-09-10T18:54:37.282 INFO:teuthology.orchestra.run.smithi119.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 185 kB/s | 38 kB 00:00 2024-09-10T18:54:37.332 INFO:teuthology.orchestra.run.smithi119.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-09-10T18:54:37.799 INFO:teuthology.orchestra.run.smithi119.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 524 kB/s | 245 kB 00:00 2024-09-10T18:54:37.802 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T18:54:37.837 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 12/12 2024-09-10T18:54:37.991 INFO:teuthology.orchestra.run.smithi119.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.6 MB/s | 3.0 MB 00:01 2024-09-10T18:54:38.108 INFO:teuthology.orchestra.run.smithi119.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-09-10T18:54:38.641 INFO:teuthology.orchestra.run.smithi119.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 3.6 MB/s | 4.9 MB 00:01 2024-09-10T18:54:39.325 INFO:teuthology.orchestra.run.smithi119.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.3 MB/s | 5.1 MB 00:01 2024-09-10T18:54:40.191 INFO:teuthology.orchestra.run.smithi119.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.8 MB/s | 16 MB 00:02 2024-09-10T18:54:40.193 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:40.193 INFO:teuthology.orchestra.run.smithi119.stdout:Total 9.7 MB/s | 38 MB 00:03 2024-09-10T18:54:40.333 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 12/12 2024-09-10T18:54:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 1/12 2024-09-10T18:54:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T18:54:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T18:54:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T18:54:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T18:54:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T18:54:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T18:54:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T18:54:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T18:54:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T18:54:40.334 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T18:54:40.551 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:54:40.578 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:54:40.578 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:54:40.743 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T18:54:40.743 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:40.744 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:54:40.744 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:40.744 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T18:54:40.744 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T18:54:40.744 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T18:54:40.744 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T18:54:40.744 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T18:54:40.744 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T18:54:40.744 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T18:54:40.744 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T18:54:40.744 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T18:54:40.744 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T18:54:40.745 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T18:54:40.745 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:40.745 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:54:41.082 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:54:41.083 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:54:41.146 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-rook 2024-09-10T18:54:41.715 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:13 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:54:41.849 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:54:41.850 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:41.850 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-10T18:54:41.850 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:41.851 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:54:41.851 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 47 k 2024-09-10T18:54:41.851 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-10T18:54:41.851 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T18:54:41.851 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T18:54:41.851 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T18:54:41.851 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T18:54:41.851 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T18:54:41.851 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T18:54:41.851 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T18:54:41.851 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T18:54:41.851 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T18:54:41.851 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T18:54:41.851 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T18:54:41.852 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T18:54:41.852 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:41.852 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:54:41.852 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:41.852 INFO:teuthology.orchestra.run.smithi194.stdout:Install 13 Packages 2024-09-10T18:54:41.852 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:41.852 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 2.2 M 2024-09-10T18:54:41.853 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 26 M 2024-09-10T18:54:41.853 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:54:41.854 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:54:41.939 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T18:54:41.995 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T18:54:42.021 INFO:teuthology.orchestra.run.smithi194.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 220 kB/s | 26 kB 00:00 2024-09-10T18:54:42.046 INFO:teuthology.orchestra.run.smithi194.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 135 kB/s | 19 kB 00:00 2024-09-10T18:54:42.071 INFO:teuthology.orchestra.run.smithi194.stdout:(3/13): ceph-mgr-rook-17.2.7-1606.g5be57395.el9 284 kB/s | 47 kB 00:00 2024-09-10T18:54:42.088 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T18:54:42.124 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T18:54:42.188 INFO:teuthology.orchestra.run.smithi194.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-10T18:54:42.213 INFO:teuthology.orchestra.run.smithi194.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 222 kB 00:00 2024-09-10T18:54:42.239 INFO:teuthology.orchestra.run.smithi194.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-10T18:54:42.264 INFO:teuthology.orchestra.run.smithi194.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 640 kB/s | 32 kB 00:00 2024-09-10T18:54:42.281 INFO:teuthology.orchestra.run.smithi194.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 352 kB/s | 14 kB 00:00 2024-09-10T18:54:42.306 INFO:teuthology.orchestra.run.smithi194.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-10T18:54:42.308 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T18:54:42.340 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T18:54:42.340 INFO:teuthology.orchestra.run.smithi194.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.9 MB/s | 220 kB 00:00 2024-09-10T18:54:42.373 INFO:teuthology.orchestra.run.smithi194.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.8 MB/s | 90 kB 00:00 2024-09-10T18:54:42.408 INFO:teuthology.orchestra.run.smithi194.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 587 kB/s | 59 kB 00:00 2024-09-10T18:54:42.433 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T18:54:42.440 INFO:teuthology.orchestra.run.smithi194.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-09-10T18:54:42.443 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:42.444 INFO:teuthology.orchestra.run.smithi194.stdout:Total 3.7 MB/s | 2.2 MB 00:00 2024-09-10T18:54:42.508 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:54:42.522 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:54:42.522 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:54:42.610 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:54:42.610 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:54:43.069 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T18:54:43.137 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:54:43.187 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T18:54:43.294 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T18:54:43.358 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T18:54:43.414 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T18:54:43.459 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T18:54:43.500 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T18:54:43.610 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T18:54:43.712 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T18:54:43.761 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T18:54:44.428 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T18:54:44.479 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T18:54:44.541 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T18:54:44.575 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T18:54:44.620 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T18:54:44.644 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 13/13 2024-09-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 13/13 2024-09-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 1/13 2024-09-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T18:54:45.413 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T18:54:45.414 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T18:54:45.414 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T18:54:45.414 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T18:54:45.414 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T18:54:45.452 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T18:54:45.480 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 12/12 2024-09-10T18:54:45.765 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T18:54:45.765 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:45.765 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:54:45.766 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:45.766 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T18:54:45.766 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T18:54:45.766 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T18:54:45.766 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T18:54:45.766 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T18:54:45.766 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T18:54:45.766 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T18:54:45.766 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T18:54:45.767 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T18:54:45.767 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T18:54:45.767 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T18:54:45.767 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T18:54:45.767 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:45.767 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:54:45.999 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T18:54:46.572 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:18 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:54:46.706 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:54:46.707 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:46.707 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-10T18:54:46.707 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:46.707 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:54:46.707 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 124 k 2024-09-10T18:54:46.707 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-10T18:54:46.707 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T18:54:46.707 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T18:54:46.707 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T18:54:46.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T18:54:46.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T18:54:46.708 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:46.708 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:54:46.708 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:46.708 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-09-10T18:54:46.708 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:46.709 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 7.0 M 2024-09-10T18:54:46.709 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 33 M 2024-09-10T18:54:46.709 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:54:47.051 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1606.g5be57395.e 641 kB/s | 124 kB 00:00 2024-09-10T18:54:47.084 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-10T18:54:47.127 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.4 MB/s | 58 kB 00:00 2024-09-10T18:54:47.151 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.1 MB/s | 74 kB 00:00 2024-09-10T18:54:47.185 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.0 MB/s | 548 kB 00:00 2024-09-10T18:54:47.327 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-09-10T18:54:47.329 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:47.329 INFO:teuthology.orchestra.run.smithi194.stdout:Total 11 MB/s | 7.0 MB 00:00 2024-09-10T18:54:47.400 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:54:47.410 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:54:47.411 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:54:47.509 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:54:47.510 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:54:47.816 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:54:47.863 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 12/12 2024-09-10T18:54:47.864 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 1/12 2024-09-10T18:54:47.864 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T18:54:47.864 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T18:54:47.864 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T18:54:47.864 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T18:54:47.864 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T18:54:47.864 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T18:54:47.864 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T18:54:47.864 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T18:54:47.864 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T18:54:47.864 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T18:54:47.992 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T18:54:48.045 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T18:54:48.356 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T18:54:48.356 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:48.356 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:54:48.356 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:48.356 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T18:54:48.357 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T18:54:48.357 INFO:teuthology.orchestra.run.smithi119.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T18:54:48.357 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T18:54:48.357 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T18:54:48.357 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T18:54:48.357 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T18:54:48.357 INFO:teuthology.orchestra.run.smithi119.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T18:54:48.357 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T18:54:48.357 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T18:54:48.357 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T18:54:48.357 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:48.357 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:54:48.560 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T18:54:48.638 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T18:54:48.703 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T18:54:48.732 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 6/6 2024-09-10T18:54:48.795 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-rook 2024-09-10T18:54:49.378 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:15 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:54:49.523 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:54:49.524 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:49.524 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-10T18:54:49.524 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:49.524 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:54:49.524 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 47 k 2024-09-10T18:54:49.524 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-10T18:54:49.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T18:54:49.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T18:54:49.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T18:54:49.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T18:54:49.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T18:54:49.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T18:54:49.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T18:54:49.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T18:54:49.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T18:54:49.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T18:54:49.526 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T18:54:49.526 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T18:54:49.526 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:49.526 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:54:49.526 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:49.526 INFO:teuthology.orchestra.run.smithi119.stdout:Install 13 Packages 2024-09-10T18:54:49.526 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:49.527 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 2.2 M 2024-09-10T18:54:49.527 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 26 M 2024-09-10T18:54:49.527 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:54:49.560 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 6/6 2024-09-10T18:54:49.560 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/6 2024-09-10T18:54:49.560 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T18:54:49.560 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T18:54:49.560 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T18:54:49.561 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T18:54:49.838 INFO:teuthology.orchestra.run.smithi119.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 169 kB/s | 19 kB 00:00 2024-09-10T18:54:49.854 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T18:54:49.854 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:49.854 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:54:49.855 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:49.855 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T18:54:49.855 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T18:54:49.855 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T18:54:49.855 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T18:54:49.855 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T18:54:49.855 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:49.855 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:54:49.863 INFO:teuthology.orchestra.run.smithi119.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 185 kB/s | 26 kB 00:00 2024-09-10T18:54:49.888 INFO:teuthology.orchestra.run.smithi119.stdout:(3/13): ceph-mgr-rook-17.2.7-1606.g5be57395.el9 290 kB/s | 47 kB 00:00 2024-09-10T18:54:49.980 INFO:teuthology.orchestra.run.smithi119.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.5 MB/s | 222 kB 00:00 2024-09-10T18:54:50.006 INFO:teuthology.orchestra.run.smithi119.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.1 MB/s | 159 kB 00:00 2024-09-10T18:54:50.039 INFO:teuthology.orchestra.run.smithi119.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 915 kB/s | 54 kB 00:00 2024-09-10T18:54:50.073 INFO:teuthology.orchestra.run.smithi119.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 481 kB/s | 32 kB 00:00 2024-09-10T18:54:50.098 INFO:teuthology.orchestra.run.smithi119.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 248 kB/s | 14 kB 00:00 2024-09-10T18:54:50.103 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-fuse 2024-09-10T18:54:50.123 INFO:teuthology.orchestra.run.smithi119.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-10T18:54:50.157 INFO:teuthology.orchestra.run.smithi119.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-10T18:54:50.182 INFO:teuthology.orchestra.run.smithi119.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-10T18:54:50.208 INFO:teuthology.orchestra.run.smithi119.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 705 kB/s | 59 kB 00:00 2024-09-10T18:54:50.250 INFO:teuthology.orchestra.run.smithi119.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-10T18:54:50.252 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:50.253 INFO:teuthology.orchestra.run.smithi119.stdout:Total 3.0 MB/s | 2.2 MB 00:00 2024-09-10T18:54:50.319 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:54:50.334 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:54:50.334 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:54:50.423 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:54:50.423 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:54:50.671 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:22 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:54:50.804 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:54:50.804 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:50.805 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-10T18:54:50.805 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:50.805 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:54:50.805 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 813 k 2024-09-10T18:54:50.805 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-09-10T18:54:50.805 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T18:54:50.805 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:50.805 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:54:50.805 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:50.805 INFO:teuthology.orchestra.run.smithi194.stdout:Install 2 Packages 2024-09-10T18:54:50.805 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:50.806 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 893 k 2024-09-10T18:54:50.806 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 2.6 M 2024-09-10T18:54:50.806 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:54:50.927 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:54:51.096 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T18:54:51.159 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T18:54:51.214 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T18:54:51.236 INFO:teuthology.orchestra.run.smithi194.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 532 kB/s | 80 kB 00:00 2024-09-10T18:54:51.251 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T18:54:51.293 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T18:54:51.395 INFO:teuthology.orchestra.run.smithi194.stdout:(2/2): ceph-fuse-17.2.7-1606.g5be57395.el9.x86_ 2.6 MB/s | 813 kB 00:00 2024-09-10T18:54:51.396 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:51.396 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.5 MB/s | 893 kB 00:00 2024-09-10T18:54:51.402 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T18:54:51.409 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:54:51.431 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:54:51.432 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:54:51.512 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T18:54:51.513 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:54:51.513 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:54:51.561 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T18:54:51.702 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:54:51.849 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T18:54:51.919 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2/2 2024-09-10T18:54:52.270 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T18:54:52.325 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T18:54:52.359 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T18:54:52.405 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T18:54:52.429 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 13/13 2024-09-10T18:54:52.649 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2/2 2024-09-10T18:54:52.649 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/2 2024-09-10T18:54:52.931 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T18:54:52.931 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:52.931 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:54:52.931 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T18:54:52.931 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:52.931 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:54:53.216 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-volume 2024-09-10T18:54:53.291 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 13/13 2024-09-10T18:54:53.292 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 1/13 2024-09-10T18:54:53.292 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T18:54:53.292 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T18:54:53.292 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T18:54:53.292 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T18:54:53.292 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T18:54:53.292 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T18:54:53.292 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T18:54:53.292 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T18:54:53.293 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T18:54:53.293 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T18:54:53.293 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T18:54:53.616 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T18:54:53.616 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:53.616 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:54:53.616 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:53.616 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T18:54:53.616 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T18:54:53.616 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T18:54:53.616 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T18:54:53.616 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T18:54:53.616 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T18:54:53.617 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T18:54:53.617 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T18:54:53.617 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T18:54:53.617 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T18:54:53.617 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T18:54:53.617 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T18:54:53.617 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:53.617 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:54:53.797 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:25 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:54:53.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T18:54:53.934 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:54:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:53.935 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-09-10T18:54:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:54:53.935 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 270 k 2024-09-10T18:54:53.935 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:53.935 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:54:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-10T18:54:53.936 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 270 k 2024-09-10T18:54:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 1.3 M 2024-09-10T18:54:53.936 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:54:54.158 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-volume-17.2.7-1606.g5be57395.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-10T18:54:54.159 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:54.159 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-10T18:54:54.159 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:54:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:54:54.167 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:54:54.195 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:54:54.196 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:54:54.402 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:54:54.447 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:20 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:54:54.485 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:54:54.501 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:54:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:54:54.501 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T18:54:54.501 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:54.580 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:54:54.581 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:54.581 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-10T18:54:54.581 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:54.581 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:54:54.581 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 124 k 2024-09-10T18:54:54.581 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-10T18:54:54.581 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T18:54:54.581 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T18:54:54.581 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T18:54:54.582 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T18:54:54.582 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T18:54:54.582 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:54.582 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:54:54.582 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:54.582 INFO:teuthology.orchestra.run.smithi119.stdout:Install 6 Packages 2024-09-10T18:54:54.582 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:54.582 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 7.0 M 2024-09-10T18:54:54.583 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 33 M 2024-09-10T18:54:54.583 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:54:55.269 INFO:teuthology.orchestra.run.smithi119.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1606.g5be57395.e 637 kB/s | 124 kB 00:00 2024-09-10T18:54:55.302 INFO:teuthology.orchestra.run.smithi119.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-10T18:54:55.352 INFO:teuthology.orchestra.run.smithi119.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-10T18:54:55.377 INFO:teuthology.orchestra.run.smithi119.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 787 kB/s | 58 kB 00:00 2024-09-10T18:54:55.402 INFO:teuthology.orchestra.run.smithi119.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.5 MB/s | 74 kB 00:00 2024-09-10T18:54:55.540 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:54:55.540 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:55.540 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:54:55.541 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:55.541 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:55.541 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:54:55.544 INFO:teuthology.orchestra.run.smithi119.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-09-10T18:54:55.546 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:55.546 INFO:teuthology.orchestra.run.smithi119.stdout:Total 7.2 MB/s | 7.0 MB 00:00 2024-09-10T18:54:55.615 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:54:55.626 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:54:55.626 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:54:55.725 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:54:55.725 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:54:55.846 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados-devel 2024-09-10T18:54:56.031 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:54:56.201 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T18:54:56.254 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T18:54:56.429 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:28 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:54:56.565 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:54:56.565 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:56.565 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-09-10T18:54:56.566 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:56.566 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:54:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:17.2.7-1606.g5be57395.el9 ceph 126 k 2024-09-10T18:54:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:56.566 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:54:56.566 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:56.566 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-10T18:54:56.566 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:56.567 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 126 k 2024-09-10T18:54:56.567 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 448 k 2024-09-10T18:54:56.567 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:54:56.772 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T18:54:56.786 INFO:teuthology.orchestra.run.smithi194.stdout:librados-devel-17.2.7-1606.g5be57395.el9.x86_64 575 kB/s | 126 kB 00:00 2024-09-10T18:54:56.787 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:56.787 INFO:teuthology.orchestra.run.smithi194.stdout:Total 570 kB/s | 126 kB 00:00 2024-09-10T18:54:56.788 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:54:56.793 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:54:56.793 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:54:56.856 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:54:56.856 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:54:56.865 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T18:54:56.920 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T18:54:56.950 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 6/6 2024-09-10T18:54:57.018 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:54:57.109 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:54:57.559 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:54:57.755 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 6/6 2024-09-10T18:54:57.756 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/6 2024-09-10T18:54:57.756 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T18:54:57.756 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T18:54:57.756 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T18:54:57.756 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T18:54:57.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:54:57.817 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:57.817 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:54:57.817 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:54:57.817 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:57.817 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:54:58.040 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T18:54:58.041 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:58.041 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:54:58.041 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:54:58.041 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T18:54:58.041 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T18:54:58.041 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T18:54:58.041 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T18:54:58.041 INFO:teuthology.orchestra.run.smithi119.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T18:54:58.041 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:58.041 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:54:58.057 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs2 2024-09-10T18:54:58.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-fuse 2024-09-10T18:54:58.645 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:30 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:54:58.739 INFO:teuthology.orchestra.run.smithi194.stdout:Package libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:54:58.784 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:54:58.785 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-10T18:54:58.785 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:54:58.840 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:24 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:54:58.873 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs-devel 2024-09-10T18:54:58.972 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:54:58.973 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:58.973 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-10T18:54:58.973 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:58.973 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:54:58.973 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 813 k 2024-09-10T18:54:58.973 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2024-09-10T18:54:58.973 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T18:54:58.973 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:58.973 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:54:58.973 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:54:58.973 INFO:teuthology.orchestra.run.smithi119.stdout:Install 2 Packages 2024-09-10T18:54:58.974 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:54:58.974 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 893 k 2024-09-10T18:54:58.974 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 2.6 M 2024-09-10T18:54:58.974 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:54:59.238 INFO:teuthology.orchestra.run.smithi119.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 550 kB/s | 80 kB 00:00 2024-09-10T18:54:59.388 INFO:teuthology.orchestra.run.smithi119.stdout:(2/2): ceph-fuse-17.2.7-1606.g5be57395.el9.x86_ 2.7 MB/s | 813 kB 00:00 2024-09-10T18:54:59.388 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:59.389 INFO:teuthology.orchestra.run.smithi119.stdout:Total 2.1 MB/s | 893 kB 00:00 2024-09-10T18:54:59.400 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:54:59.421 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:54:59.421 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:54:59.442 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:31 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:54:59.500 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:54:59.500 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:54:59.575 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:54:59.575 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:59.576 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-09-10T18:54:59.576 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:59.576 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:54:59.576 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:17.2.7-1606.g5be57395.el9 ceph 25 k 2024-09-10T18:54:59.576 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:59.576 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:54:59.576 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:54:59.576 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-10T18:54:59.576 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:54:59.577 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 25 k 2024-09-10T18:54:59.577 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 110 k 2024-09-10T18:54:59.577 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:54:59.682 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:54:59.724 INFO:teuthology.orchestra.run.smithi194.stdout:libcephfs-devel-17.2.7-1606.g5be57395.el9.x86_6 169 kB/s | 25 kB 00:00 2024-09-10T18:54:59.725 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:54:59.725 INFO:teuthology.orchestra.run.smithi194.stdout:Total 168 kB/s | 25 kB 00:00 2024-09-10T18:54:59.725 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:54:59.729 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:54:59.729 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:54:59.730 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:54:59.730 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:54:59.810 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T18:54:59.812 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:54:59.878 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2/2 2024-09-10T18:54:59.921 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:00.270 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:00.493 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:00.493 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:00.493 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:55:00.493 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:55:00.493 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:00.493 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:55:00.571 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2/2 2024-09-10T18:55:00.571 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/2 2024-09-10T18:55:00.672 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados2 2024-09-10T18:55:00.852 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T18:55:00.853 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:00.853 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:55:00.853 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T18:55:00.853 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:00.853 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:55:01.110 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-volume 2024-09-10T18:55:01.239 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:33 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:55:01.329 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:55:01.371 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:55:01.372 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-10T18:55:01.373 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:55:01.438 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librbd1 2024-09-10T18:55:01.686 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:27 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:55:01.821 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:55:01.822 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:01.822 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2024-09-10T18:55:01.822 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:01.822 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:55:01.822 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 270 k 2024-09-10T18:55:01.823 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:01.823 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:55:01.823 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:01.823 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-10T18:55:01.823 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:01.823 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 270 k 2024-09-10T18:55:01.823 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 1.3 M 2024-09-10T18:55:01.823 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:55:02.010 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:34 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:55:02.052 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-volume-17.2.7-1606.g5be57395.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-10T18:55:02.052 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:55:02.052 INFO:teuthology.orchestra.run.smithi119.stdout:Total 1.1 MB/s | 270 kB 00:00 2024-09-10T18:55:02.053 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:55:02.061 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:55:02.061 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:55:02.087 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:55:02.087 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:55:02.101 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:55:02.144 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:55:02.145 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-10T18:55:02.145 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:55:02.205 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rados 2024-09-10T18:55:02.299 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:55:02.370 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:55:02.387 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:55:02.387 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:55:02.387 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T18:55:02.387 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:02.767 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:34 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:55:02.857 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:55:02.899 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:55:02.901 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-10T18:55:02.901 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:55:02.962 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rgw 2024-09-10T18:55:03.416 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T18:55:03.417 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:03.417 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:55:03.417 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T18:55:03.417 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:03.417 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:55:03.535 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:35 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:55:03.625 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:55:03.668 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:55:03.669 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-10T18:55:03.669 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:55:03.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados-devel 2024-09-10T18:55:03.733 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-cephfs 2024-09-10T18:55:04.279 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:30 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:55:04.300 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:36 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:55:04.391 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:55:04.415 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:55:04.415 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:04.415 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-09-10T18:55:04.416 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:04.416 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:55:04.416 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:17.2.7-1606.g5be57395.el9 ceph 126 k 2024-09-10T18:55:04.416 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:04.416 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:55:04.416 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:04.416 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-10T18:55:04.416 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:04.416 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 126 k 2024-09-10T18:55:04.416 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 448 k 2024-09-10T18:55:04.416 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:55:04.433 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:55:04.434 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-10T18:55:04.434 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:55:04.499 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rbd 2024-09-10T18:55:04.640 INFO:teuthology.orchestra.run.smithi119.stdout:librados-devel-17.2.7-1606.g5be57395.el9.x86_64 564 kB/s | 126 kB 00:00 2024-09-10T18:55:04.641 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:55:04.641 INFO:teuthology.orchestra.run.smithi119.stdout:Total 560 kB/s | 126 kB 00:00 2024-09-10T18:55:04.641 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:55:04.646 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:55:04.647 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:55:04.712 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:55:04.713 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:55:04.877 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:55:04.988 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:05.061 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:55:05.150 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:55:05.193 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:55:05.194 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-09-10T18:55:05.194 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:55:05.271 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-fuse 2024-09-10T18:55:05.417 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:05.679 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:05.680 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:05.680 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:55:05.680 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:55:05.680 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:05.680 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:55:05.837 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:55:05.941 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs2 2024-09-10T18:55:05.970 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:55:05.970 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:55:05.970 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-09-10T18:55:05.971 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:55:05.971 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:55:05.971 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 82 k 2024-09-10T18:55:05.971 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:05.971 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:55:05.971 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:55:05.971 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-10T18:55:05.971 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:05.972 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 82 k 2024-09-10T18:55:05.972 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 222 k 2024-09-10T18:55:05.972 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:55:06.152 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-fuse-17.2.7-1606.g5be57395.el9.x86_64.rpm 453 kB/s | 82 kB 00:00 2024-09-10T18:55:06.153 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:55:06.153 INFO:teuthology.orchestra.run.smithi194.stdout:Total 450 kB/s | 82 kB 00:00 2024-09-10T18:55:06.153 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:55:06.160 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:55:06.160 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:55:06.224 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:55:06.224 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:55:06.369 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:55:06.466 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:06.513 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:32 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:55:06.605 INFO:teuthology.orchestra.run.smithi119.stdout:Package libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:55:06.647 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:55:06.648 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-10T18:55:06.649 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:55:06.724 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install libcephfs-devel 2024-09-10T18:55:06.835 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:07.102 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:07.102 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:07.102 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:55:07.102 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:55:07.102 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:07.102 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:55:07.334 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:33 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:55:07.394 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-mirror 2024-09-10T18:55:07.467 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:55:07.468 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:07.468 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-09-10T18:55:07.468 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:07.468 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:55:07.468 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:17.2.7-1606.g5be57395.el9 ceph 25 k 2024-09-10T18:55:07.468 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:07.468 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:55:07.468 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:07.468 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-10T18:55:07.468 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:07.468 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 25 k 2024-09-10T18:55:07.468 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 110 k 2024-09-10T18:55:07.469 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:55:07.627 INFO:teuthology.orchestra.run.smithi119.stdout:libcephfs-devel-17.2.7-1606.g5be57395.el9.x86_6 157 kB/s | 25 kB 00:00 2024-09-10T18:55:07.628 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:55:07.628 INFO:teuthology.orchestra.run.smithi119.stdout:Total 156 kB/s | 25 kB 00:00 2024-09-10T18:55:07.628 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:55:07.632 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:55:07.632 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:55:07.633 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:55:07.633 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:55:07.710 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:55:07.817 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:07.976 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:39 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:55:08.110 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:55:08.110 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:55:08.110 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-09-10T18:55:08.111 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:55:08.111 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:55:08.111 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.9 M 2024-09-10T18:55:08.111 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:08.111 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:55:08.111 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:55:08.111 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-10T18:55:08.111 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:08.111 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 2.9 M 2024-09-10T18:55:08.111 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 12 M 2024-09-10T18:55:08.111 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:55:08.195 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:08.437 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:08.438 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:08.438 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:55:08.438 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:55:08.438 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:08.438 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:55:08.496 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-mirror-17.2.7-1606.g5be57395.el9.x86_64.rpm 7.6 MB/s | 2.9 MB 00:00 2024-09-10T18:55:08.497 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:55:08.497 INFO:teuthology.orchestra.run.smithi194.stdout:Total 7.6 MB/s | 2.9 MB 00:00 2024-09-10T18:55:08.497 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:55:08.506 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:55:08.506 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:55:08.586 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:55:08.586 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:55:08.609 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librados2 2024-09-10T18:55:09.008 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:55:09.072 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:09.097 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:09.097 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:55:09.098 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T18:55:09.098 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T18:55:09.098 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T18:55:09.098 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:09.173 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:35 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:55:09.262 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:55:09.304 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:55:09.305 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-10T18:55:09.305 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:55:09.380 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install librbd1 2024-09-10T18:55:09.945 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:35 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:55:10.035 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:55:10.077 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:55:10.078 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-10T18:55:10.078 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:55:10.155 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rados 2024-09-10T18:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:10.203 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:55:10.433 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-nbd 2024-09-10T18:55:10.713 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:36 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:55:10.803 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:55:10.846 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:55:10.847 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-10T18:55:10.847 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:55:10.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rgw 2024-09-10T18:55:11.008 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 06:53:28 PM UTC. 2024-09-10T18:55:11.145 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-09-10T18:55:11.145 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:55:11.145 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-09-10T18:55:11.146 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:55:11.146 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-09-10T18:55:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 165 k 2024-09-10T18:55:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:11.146 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-09-10T18:55:11.146 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-09-10T18:55:11.146 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-09-10T18:55:11.146 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:11.146 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 165 k 2024-09-10T18:55:11.146 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 477 k 2024-09-10T18:55:11.146 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-09-10T18:55:11.350 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-nbd-17.2.7-1606.g5be57395.el9.x86_64.rpm 811 kB/s | 165 kB 00:00 2024-09-10T18:55:11.351 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:55:11.351 INFO:teuthology.orchestra.run.smithi194.stdout:Total 805 kB/s | 165 kB 00:00 2024-09-10T18:55:11.351 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-09-10T18:55:11.358 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-09-10T18:55:11.358 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-09-10T18:55:11.422 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-09-10T18:55:11.423 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-09-10T18:55:11.489 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:55:11.578 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-09-10T18:55:11.578 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:55:11.620 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:55:11.621 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-10T18:55:11.622 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:55:11.690 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:11.700 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-cephfs 2024-09-10T18:55:12.097 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:12.267 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:38 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:55:12.357 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:55:12.366 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:12.366 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:12.366 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-09-10T18:55:12.366 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:55:12.366 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:55:12.366 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-09-10T18:55:12.400 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:55:12.401 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-10T18:55:12.401 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:55:12.470 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install python3-rbd 2024-09-10T18:55:12.614 DEBUG:teuthology.parallel:result is None 2024-09-10T18:55:13.039 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:39 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:55:13.128 INFO:teuthology.orchestra.run.smithi119.stdout:Package python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T18:55:13.170 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:55:13.172 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2024-09-10T18:55:13.172 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:55:13.251 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-fuse 2024-09-10T18:55:13.819 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:39 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:55:13.953 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:55:13.953 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:13.953 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-09-10T18:55:13.953 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:13.954 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:55:13.954 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 82 k 2024-09-10T18:55:13.954 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:13.954 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:55:13.954 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:13.954 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-10T18:55:13.954 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:13.954 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 82 k 2024-09-10T18:55:13.954 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 222 k 2024-09-10T18:55:13.954 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:55:14.156 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-fuse-17.2.7-1606.g5be57395.el9.x86_64.rpm 405 kB/s | 82 kB 00:00 2024-09-10T18:55:14.157 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:55:14.157 INFO:teuthology.orchestra.run.smithi119.stdout:Total 403 kB/s | 82 kB 00:00 2024-09-10T18:55:14.157 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:55:14.163 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:55:14.163 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:55:14.228 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:55:14.228 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:55:14.376 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:55:14.477 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:14.907 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:15.161 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:15.161 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:15.161 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:55:15.161 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:55:15.161 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:15.161 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:55:15.382 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-mirror 2024-09-10T18:55:15.967 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:41 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:55:16.105 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:55:16.106 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:16.106 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repo Size 2024-09-10T18:55:16.106 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:16.107 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:55:16.107 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.9 M 2024-09-10T18:55:16.107 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:16.107 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:55:16.107 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:16.107 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-10T18:55:16.107 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:16.107 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 2.9 M 2024-09-10T18:55:16.107 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 12 M 2024-09-10T18:55:16.107 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:55:16.468 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-mirror-17.2.7-1606.g5be57395.el9.x86_64.rpm 8.1 MB/s | 2.9 MB 00:00 2024-09-10T18:55:16.469 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:55:16.469 INFO:teuthology.orchestra.run.smithi119.stdout:Total 8.1 MB/s | 2.9 MB 00:00 2024-09-10T18:55:16.469 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:55:16.478 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:55:16.478 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:55:16.558 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:55:16.559 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:55:16.995 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:55:17.061 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:17.086 INFO:teuthology.orchestra.run.smithi119.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T18:55:17.086 INFO:teuthology.orchestra.run.smithi119.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T18:55:17.086 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T18:55:17.086 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T18:55:17.086 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:18.146 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:18.147 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:18.147 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:55:18.147 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:55:18.147 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:18.147 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:55:18.392 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install rbd-nbd 2024-09-10T18:55:18.967 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:44 ago on Tue 10 Sep 2024 06:53:34 PM UTC. 2024-09-10T18:55:19.104 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2024-09-10T18:55:19.105 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:19.105 INFO:teuthology.orchestra.run.smithi119.stdout: Package Architecture Version Repository Size 2024-09-10T18:55:19.105 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:19.105 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2024-09-10T18:55:19.105 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 165 k 2024-09-10T18:55:19.105 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:19.105 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2024-09-10T18:55:19.105 INFO:teuthology.orchestra.run.smithi119.stdout:================================================================================ 2024-09-10T18:55:19.106 INFO:teuthology.orchestra.run.smithi119.stdout:Install 1 Package 2024-09-10T18:55:19.106 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:19.106 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 165 k 2024-09-10T18:55:19.106 INFO:teuthology.orchestra.run.smithi119.stdout:Installed size: 477 k 2024-09-10T18:55:19.106 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2024-09-10T18:55:19.314 INFO:teuthology.orchestra.run.smithi119.stdout:rbd-nbd-17.2.7-1606.g5be57395.el9.x86_64.rpm 794 kB/s | 165 kB 00:00 2024-09-10T18:55:19.314 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:55:19.315 INFO:teuthology.orchestra.run.smithi119.stdout:Total 788 kB/s | 165 kB 00:00 2024-09-10T18:55:19.315 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2024-09-10T18:55:19.322 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2024-09-10T18:55:19.323 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2024-09-10T18:55:19.390 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2024-09-10T18:55:19.390 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2024-09-10T18:55:19.555 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2024-09-10T18:55:19.689 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:20.148 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:20.414 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T18:55:20.414 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:20.414 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2024-09-10T18:55:20.414 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T18:55:20.414 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:55:20.414 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2024-09-10T18:55:20.634 DEBUG:teuthology.parallel:result is None 2024-09-10T18:55:20.635 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:55:20.777 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T18:55:20.799 INFO:teuthology.orchestra.run.smithi119.stdout:17.2.7-1606.g5be57395.el9 2024-09-10T18:55:20.799 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1606.g5be57395.el9 2024-09-10T18:55:20.799 INFO:teuthology.task.install:The correct ceph version 17.2.7-1606.g5be57395 is installed. 2024-09-10T18:55:20.801 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:55:20.947 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T18:55:20.973 INFO:teuthology.orchestra.run.smithi194.stdout:17.2.7-1606.g5be57395.el9 2024-09-10T18:55:20.973 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1606.g5be57395.el9 2024-09-10T18:55:20.973 INFO:teuthology.task.install:The correct ceph version 17.2.7-1606.g5be57395 is installed. 2024-09-10T18:55:20.975 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T18:55:20.975 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T18:55:20.975 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T18:55:21.001 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-10T18:55:21.001 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T18:55:21.043 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T18:55:21.043 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T18:55:21.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T18:55:21.067 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T18:55:21.132 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-10T18:55:21.132 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T18:55:21.161 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T18:55:21.229 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T18:55:21.229 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T18:55:21.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T18:55:21.254 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T18:55:21.319 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-10T18:55:21.319 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T18:55:21.346 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T18:55:21.413 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T18:55:21.413 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T18:55:21.413 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T18:55:21.438 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T18:55:21.503 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-10T18:55:21.503 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T18:55:21.528 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T18:55:21.593 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-10T18:55:21.678 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588', 'cephadm_mode': 'cephadm-package'} 2024-09-10T18:55:21.678 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:55:21.679 INFO:tasks.cephadm:Cluster fsid is 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:55:21.679 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-10T18:55:21.679 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.119', 'mon.c': '[v2:172.21.15.119:3301,v1:172.21.15.119:6790]', 'mon.b': '172.21.15.194'} 2024-09-10T18:55:21.679 INFO:tasks.cephadm:First mon is mon.a on smithi119 2024-09-10T18:55:21.679 INFO:tasks.cephadm:First mgr is y 2024-09-10T18:55:21.680 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-10T18:55:21.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo hostname $(hostname -s) 2024-09-10T18:55:21.705 DEBUG:teuthology.orchestra.run.smithi194:> sudo hostname $(hostname -s) 2024-09-10T18:55:21.760 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 on all hosts... 2024-09-10T18:55:21.760 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 pull 2024-09-10T18:55:21.762 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 pull 2024-09-10T18:55:23.360 INFO:teuthology.orchestra.run.smithi194.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T18:55:23.488 INFO:teuthology.orchestra.run.smithi119.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T18:56:15.354 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-09-10T18:56:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_version": "ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)", 2024-09-10T18:56:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: "image_id": "4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9", 2024-09-10T18:56:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: "repo_digests": [ 2024-09-10T18:56:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16" 2024-09-10T18:56:15.355 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-09-10T18:56:15.356 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-09-10T18:56:17.002 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-10T18:56:17.002 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)", 2024-09-10T18:56:17.002 INFO:teuthology.orchestra.run.smithi119.stdout: "image_id": "4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9", 2024-09-10T18:56:17.002 INFO:teuthology.orchestra.run.smithi119.stdout: "repo_digests": [ 2024-09-10T18:56:17.002 INFO:teuthology.orchestra.run.smithi119.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16" 2024-09-10T18:56:17.002 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-10T18:56:17.002 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-10T18:56:17.021 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph 2024-09-10T18:56:17.071 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph 2024-09-10T18:56:17.105 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /etc/ceph 2024-09-10T18:56:17.130 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /etc/ceph 2024-09-10T18:56:17.174 INFO:tasks.cephadm:Writing seed config... 2024-09-10T18:56:17.175 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-10T18:56:17.175 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-10T18:56:17.175 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-09-10T18:56:17.175 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-09-10T18:56:17.175 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-09-10T18:56:17.175 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-09-10T18:56:17.176 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-10T18:56:17.176 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-10T18:56:17.176 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-10T18:56:17.176 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-10T18:56:17.176 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-09-10T18:56:17.176 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-10T18:56:17.176 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-10T18:56:17.176 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-09-10T18:56:17.176 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-09-10T18:56:17.176 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-10T18:56:17.177 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T18:56:17.177 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-10T18:56:17.192 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 3afe9804-6fa6-11ef-bcea-c7b262605968 mon election default strategy = 1 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-09-10T18:56:17.192 DEBUG:teuthology.orchestra.run.smithi119:mon.a> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.a.service 2024-09-10T18:56:17.235 DEBUG:teuthology.orchestra.run.smithi119:mgr.y> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mgr.y.service 2024-09-10T18:56:17.277 INFO:tasks.cephadm:Bootstrapping... 2024-09-10T18:56:17.277 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 -v bootstrap --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.119 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-10T18:56:17.474 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:56:17.474 INFO:teuthology.orchestra.run.smithi119.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588', '-v', 'bootstrap', '--fsid', '3afe9804-6fa6-11ef-bcea-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.119', '--skip-admin-label'] 2024-09-10T18:56:17.497 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stdout 5.2.2 2024-09-10T18:56:17.497 INFO:teuthology.orchestra.run.smithi119.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-10T18:56:17.498 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying podman|docker is present... 2024-09-10T18:56:17.528 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stdout 5.2.2 2024-09-10T18:56:17.528 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying lvm2 is present... 2024-09-10T18:56:17.529 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying time synchronization is in place... 2024-09-10T18:56:17.536 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-10T18:56:17.536 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-10T18:56:17.543 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-10T18:56:17.543 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-09-10T18:56:17.549 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout enabled 2024-09-10T18:56:17.556 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout active 2024-09-10T18:56:17.556 INFO:teuthology.orchestra.run.smithi119.stdout:Unit chronyd.service is enabled and running 2024-09-10T18:56:17.556 INFO:teuthology.orchestra.run.smithi119.stdout:Repeating the final host check... 2024-09-10T18:56:17.578 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stdout 5.2.2 2024-09-10T18:56:17.578 INFO:teuthology.orchestra.run.smithi119.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-10T18:56:17.578 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl is present 2024-09-10T18:56:17.578 INFO:teuthology.orchestra.run.smithi119.stdout:lvcreate is present 2024-09-10T18:56:17.585 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-10T18:56:17.585 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-10T18:56:17.591 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-10T18:56:17.591 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-09-10T18:56:17.598 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout enabled 2024-09-10T18:56:17.606 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout active 2024-09-10T18:56:17.606 INFO:teuthology.orchestra.run.smithi119.stdout:Unit chronyd.service is enabled and running 2024-09-10T18:56:17.606 INFO:teuthology.orchestra.run.smithi119.stdout:Host looks OK 2024-09-10T18:56:17.606 INFO:teuthology.orchestra.run.smithi119.stdout:Cluster fsid: 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:56:17.607 INFO:teuthology.orchestra.run.smithi119.stdout:Acquiring lock 139868605723408 on /run/cephadm/3afe9804-6fa6-11ef-bcea-c7b262605968.lock 2024-09-10T18:56:17.607 INFO:teuthology.orchestra.run.smithi119.stdout:Lock 139868605723408 acquired on /run/cephadm/3afe9804-6fa6-11ef-bcea-c7b262605968.lock 2024-09-10T18:56:17.607 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying IP 172.21.15.119 port 3300 ... 2024-09-10T18:56:17.607 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying IP 172.21.15.119 port 6789 ... 2024-09-10T18:56:17.607 INFO:teuthology.orchestra.run.smithi119.stdout:Base mon IP(s) is [172.21.15.119:3300, 172.21.15.119:6789], mon addrv is [v2:172.21.15.119:3300,v1:172.21.15.119:6789] 2024-09-10T18:56:17.610 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.119 metric 100 2024-09-10T18:56:17.610 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.119 metric 100 2024-09-10T18:56:17.613 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-10T18:56:17.613 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-10T18:56:17.613 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-10T18:56:17.615 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-10T18:56:17.615 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-10T18:56:17.615 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-10T18:56:17.616 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-10T18:56:17.616 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link noprefixroute 2024-09-10T18:56:17.616 INFO:teuthology.orchestra.run.smithi119.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-10T18:56:17.617 INFO:teuthology.orchestra.run.smithi119.stdout:Mon IP `172.21.15.119` is in CIDR network `172.21.0.0/20` 2024-09-10T18:56:17.617 INFO:teuthology.orchestra.run.smithi119.stdout:Mon IP `172.21.15.119` is in CIDR network `172.21.0.0/20` 2024-09-10T18:56:17.617 INFO:teuthology.orchestra.run.smithi119.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-10T18:56:17.617 INFO:teuthology.orchestra.run.smithi119.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-10T18:56:17.617 INFO:teuthology.orchestra.run.smithi119.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T18:56:18.020 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stdout 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 2024-09-10T18:56:18.020 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T18:56:18.020 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Getting image source signatures 2024-09-10T18:56:18.020 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Copying blob sha256:5df13064c03763eb21a89b455ab82ba4ec435f51a3f340cae76514aaf2996d4c 2024-09-10T18:56:18.020 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-10T18:56:18.020 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Copying config sha256:4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 2024-09-10T18:56:18.021 INFO:teuthology.orchestra.run.smithi119.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-10T18:56:19.026 INFO:teuthology.orchestra.run.smithi119.stdout:ceph: stdout ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable) 2024-09-10T18:56:19.026 INFO:teuthology.orchestra.run.smithi119.stdout:Ceph version: ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable) 2024-09-10T18:56:19.026 INFO:teuthology.orchestra.run.smithi119.stdout:Extracting ceph user uid/gid from container image... 2024-09-10T18:56:19.996 INFO:teuthology.orchestra.run.smithi119.stdout:stat: stdout 167 167 2024-09-10T18:56:19.997 INFO:teuthology.orchestra.run.smithi119.stdout:Creating initial keys... 2024-09-10T18:56:21.061 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph-authtool: stdout AQBUluBmZ/uJFRAATlSSL1bf9JmWz6EYdwuqHg== 2024-09-10T18:56:22.133 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph-authtool: stdout AQBVluBmBP1SGRAA65KnR6tnobNmw+52fsnffA== 2024-09-10T18:56:23.047 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph-authtool: stdout AQBWluBmQMctHBAAQYw3tkEqb4dngnGVFJC9qQ== 2024-09-10T18:56:23.048 INFO:teuthology.orchestra.run.smithi119.stdout:Creating initial monmap... 2024-09-10T18:56:24.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-10T18:56:24.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-09-10T18:56:24.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:56:24.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-10T18:56:24.038 INFO:teuthology.orchestra.run.smithi119.stdout:monmaptool for a [v2:172.21.15.119:3300,v1:172.21.15.119:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-10T18:56:24.038 INFO:teuthology.orchestra.run.smithi119.stdout:setting min_mon_release = octopus 2024-09-10T18:56:24.038 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: set fsid to 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:56:24.039 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-10T18:56:24.039 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:56:24.039 INFO:teuthology.orchestra.run.smithi119.stdout:Creating mon... 2024-09-10T18:56:24.960 INFO:teuthology.orchestra.run.smithi119.stdout:create mon.a on 2024-09-10T18:56:25.166 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-10T18:56:25.312 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-10T18:56:25.458 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-3afe9804-6fa6-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-3afe9804-6fa6-11ef-bcea-c7b262605968.target. 2024-09-10T18:56:25.458 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-3afe9804-6fa6-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-3afe9804-6fa6-11ef-bcea-c7b262605968.target. 2024-09-10T18:56:25.628 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.a 2024-09-10T18:56:25.629 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.a.service: Unit ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.a.service not loaded. 2024-09-10T18:56:25.793 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3afe9804-6fa6-11ef-bcea-c7b262605968.target.wants/ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.a.service → /etc/systemd/system/ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@.service. 2024-09-10T18:56:26.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:26 smithi119 podman[29069]: 2024-09-10 18:56:25.93305155 +0000 UTC m=+0.015668478 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:56:26.362 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:26 smithi119 podman[29069]: 2024-09-10 18:56:26.067169828 +0000 UTC m=+0.149786746 container create edd7a8c744e9ed1922fb0f3952e9776a29bf110b276be69c6471c90030999618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD) 2024-09-10T18:56:26.362 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:26 smithi119 podman[29069]: 2024-09-10 18:56:26.261273307 +0000 UTC m=+0.343890226 container init edd7a8c744e9ed1922fb0f3952e9776a29bf110b276be69c6471c90030999618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2) 2024-09-10T18:56:26.362 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:26 smithi119 podman[29069]: 2024-09-10 18:56:26.266252078 +0000 UTC m=+0.348868997 container start edd7a8c744e9ed1922fb0f3952e9776a29bf110b276be69c6471c90030999618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-10T18:56:26.362 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:26 smithi119 ceph-mon[29107]: mkfs 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:56:26.362 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:26 smithi119 ceph-mon[29107]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-10T18:56:26.362 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:26 smithi119 bash[29069]: edd7a8c744e9ed1922fb0f3952e9776a29bf110b276be69c6471c90030999618 2024-09-10T18:56:26.376 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T18:56:26.376 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout disabled 2024-09-10T18:56:26.383 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T18:56:26.383 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-09-10T18:56:26.383 INFO:teuthology.orchestra.run.smithi119.stdout:firewalld.service is not enabled 2024-09-10T18:56:26.384 INFO:teuthology.orchestra.run.smithi119.stdout:Not possible to enable service . firewalld.service is not available 2024-09-10T18:56:26.384 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mon to start... 2024-09-10T18:56:26.384 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mon... 2024-09-10T18:56:26.641 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:26 smithi119 systemd[1]: Started Ceph mon.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T18:56:27.486 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:27 smithi119 ceph-mon[29107]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-10T18:56:27.486 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:27 smithi119 ceph-mon[29107]: monmap e1: 1 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:56:27.486 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:27 smithi119 ceph-mon[29107]: fsmap 2024-09-10T18:56:27.486 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:27 smithi119 ceph-mon[29107]: osdmap e1: 0 total, 0 up, 0 in 2024-09-10T18:56:27.486 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:27 smithi119 ceph-mon[29107]: mgrmap e1: no daemons active 2024-09-10T18:56:27.486 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:27 smithi119 ceph-mon[29107]: from='client.? 172.21.15.119:0/1500501644' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-10T18:56:27.493 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout cluster: 2024-09-10T18:56:27.493 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout id: 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:56:27.494 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-10T18:56:27.494 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-10T18:56:27.494 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout services: 2024-09-10T18:56:27.494 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.608433s) 2024-09-10T18:56:27.494 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-10T18:56:27.494 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-10T18:56:27.494 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-10T18:56:27.494 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout data: 2024-09-10T18:56:27.494 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-10T18:56:27.494 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-10T18:56:27.494 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-10T18:56:27.494 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout pgs: 2024-09-10T18:56:27.495 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-10T18:56:27.495 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.841+0000 7f38d8c3b640 1 Processor -- start 2024-09-10T18:56:27.495 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.841+0000 7f38d8c3b640 1 -- start start 2024-09-10T18:56:27.495 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.841+0000 7f38d8c3b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d40ff460 0x7f38d40ff870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:27.495 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.841+0000 7f38d8c3b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d40ffdb0 con 0x7f38d40ff460 2024-09-10T18:56:27.495 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.841+0000 7f38d259c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d40ff460 0x7f38d40ff870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:27.495 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.842+0000 7f38d259c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d40ff460 0x7f38d40ff870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45932/0 (socket says 172.21.15.119:45932) 2024-09-10T18:56:27.495 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.842+0000 7f38d259c640 1 -- 172.21.15.119:0/838118552 learned_addr learned my addr 172.21.15.119:0/838118552 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:27.495 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.842+0000 7f38d259c640 1 -- 172.21.15.119:0/838118552 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d40ffef0 con 0x7f38d40ff460 2024-09-10T18:56:27.495 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.844+0000 7f38d259c640 1 --2- 172.21.15.119:0/838118552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d40ff460 0x7f38d40ff870 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f38c400baf0 tx=0x7f38c400b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f72faa3dd7e8bf9c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:27.495 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.844+0000 7f38d159a640 1 -- 172.21.15.119:0/838118552 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38c4032640 con 0x7f38d40ff460 2024-09-10T18:56:27.495 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.844+0000 7f38d159a640 1 -- 172.21.15.119:0/838118552 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c40327a0 con 0x7f38d40ff460 2024-09-10T18:56:27.496 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.844+0000 7f38d159a640 1 -- 172.21.15.119:0/838118552 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38c4032ab0 con 0x7f38d40ff460 2024-09-10T18:56:27.496 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.845+0000 7f38d8c3b640 1 -- 172.21.15.119:0/838118552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d40ff460 msgr2=0x7f38d40ff870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:27.496 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.845+0000 7f38d8c3b640 1 --2- 172.21.15.119:0/838118552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d40ff460 0x7f38d40ff870 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f38c400baf0 tx=0x7f38c400b460 comp rx=0 tx=0).stop 2024-09-10T18:56:27.496 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.845+0000 7f38d8c3b640 1 -- 172.21.15.119:0/838118552 shutdown_connections 2024-09-10T18:56:27.496 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.845+0000 7f38d8c3b640 1 --2- 172.21.15.119:0/838118552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d40ff460 0x7f38d40ff870 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:27.496 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.845+0000 7f38d8c3b640 1 -- 172.21.15.119:0/838118552 >> 172.21.15.119:0/838118552 conn(0x7f38d40fac60 msgr2=0x7f38d40fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:27.496 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.845+0000 7f38d8c3b640 1 -- 172.21.15.119:0/838118552 shutdown_connections 2024-09-10T18:56:27.496 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.845+0000 7f38d8c3b640 1 -- 172.21.15.119:0/838118552 wait complete. 2024-09-10T18:56:27.496 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.845+0000 7f38d8c3b640 1 Processor -- start 2024-09-10T18:56:27.496 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.845+0000 7f38d8c3b640 1 -- start start 2024-09-10T18:56:27.496 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.846+0000 7f38d8c3b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d40ff460 0x7f38d4110c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:27.497 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.846+0000 7f38d8c3b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d4111180 con 0x7f38d40ff460 2024-09-10T18:56:27.497 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.846+0000 7f38d259c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d40ff460 0x7f38d4110c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:27.497 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.846+0000 7f38d259c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d40ff460 0x7f38d4110c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45942/0 (socket says 172.21.15.119:45942) 2024-09-10T18:56:27.497 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.846+0000 7f38d259c640 1 -- 172.21.15.119:0/1500501644 learned_addr learned my addr 172.21.15.119:0/1500501644 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:27.497 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.846+0000 7f38d259c640 1 -- 172.21.15.119:0/1500501644 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c400b700 con 0x7f38d40ff460 2024-09-10T18:56:27.497 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.846+0000 7f38d259c640 1 --2- 172.21.15.119:0/1500501644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d40ff460 0x7f38d4110c40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f38c400baf0 tx=0x7f38c403a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:27.497 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.846+0000 7f38c2ffd640 1 -- 172.21.15.119:0/1500501644 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38c403b8e0 con 0x7f38d40ff460 2024-09-10T18:56:27.497 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.846+0000 7f38c2ffd640 1 -- 172.21.15.119:0/1500501644 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c403ba40 con 0x7f38d40ff460 2024-09-10T18:56:27.497 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.846+0000 7f38c2ffd640 1 -- 172.21.15.119:0/1500501644 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f38c403bd10 con 0x7f38d40ff460 2024-09-10T18:56:27.498 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.846+0000 7f38d8c3b640 1 -- 172.21.15.119:0/1500501644 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d4111380 con 0x7f38d40ff460 2024-09-10T18:56:27.498 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.846+0000 7f38d8c3b640 1 -- 172.21.15.119:0/1500501644 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d4111790 con 0x7f38d40ff460 2024-09-10T18:56:27.498 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.847+0000 7f38c2ffd640 1 -- 172.21.15.119:0/1500501644 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f38c403c430 con 0x7f38d40ff460 2024-09-10T18:56:27.498 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.847+0000 7f38c2ffd640 1 -- 172.21.15.119:0/1500501644 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f38c4042070 con 0x7f38d40ff460 2024-09-10T18:56:27.498 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.847+0000 7f38d8c3b640 1 -- 172.21.15.119:0/1500501644 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d404b720 con 0x7f38d40ff460 2024-09-10T18:56:27.498 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.849+0000 7f38c2ffd640 1 -- 172.21.15.119:0/1500501644 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f38c404b060 con 0x7f38d40ff460 2024-09-10T18:56:27.498 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.905+0000 7f38d8c3b640 1 -- 172.21.15.119:0/1500501644 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f38d41007f0 con 0x7f38d40ff460 2024-09-10T18:56:27.498 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.906+0000 7f38c2ffd640 1 -- 172.21.15.119:0/1500501644 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f38c4036020 con 0x7f38d40ff460 2024-09-10T18:56:27.498 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.906+0000 7f38d8c3b640 1 -- 172.21.15.119:0/1500501644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d40ff460 msgr2=0x7f38d4110c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:27.498 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.906+0000 7f38d8c3b640 1 --2- 172.21.15.119:0/1500501644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d40ff460 0x7f38d4110c40 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f38c400baf0 tx=0x7f38c403a000 comp rx=0 tx=0).stop 2024-09-10T18:56:27.498 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.907+0000 7f38d8c3b640 1 -- 172.21.15.119:0/1500501644 shutdown_connections 2024-09-10T18:56:27.498 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.907+0000 7f38d8c3b640 1 --2- 172.21.15.119:0/1500501644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d40ff460 0x7f38d4110c40 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:27.498 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.907+0000 7f38d8c3b640 1 -- 172.21.15.119:0/1500501644 >> 172.21.15.119:0/1500501644 conn(0x7f38d40fac60 msgr2=0x7f38d40fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:27.499 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.907+0000 7f38d8c3b640 1 -- 172.21.15.119:0/1500501644 shutdown_connections 2024-09-10T18:56:27.499 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:26.907+0000 7f38d8c3b640 1 -- 172.21.15.119:0/1500501644 wait complete. 2024-09-10T18:56:27.499 INFO:teuthology.orchestra.run.smithi119.stdout:mon is available 2024-09-10T18:56:27.499 INFO:teuthology.orchestra.run.smithi119.stdout:Assimilating anything we can from ceph.conf... 2024-09-10T18:56:28.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:28 smithi119 ceph-mon[29107]: from='client.? 172.21.15.119:0/2159574148' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-10T18:56:28.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:28 smithi119 ceph-mon[29107]: from='client.? 172.21.15.119:0/2159574148' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-10T18:56:28.569 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-10T18:56:28.575 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [global] 2024-09-10T18:56:28.575 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout fsid = 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:56:28.575 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-10T18:56:28.575 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.119:3300,v1:172.21.15.119:6789] 2024-09-10T18:56:28.575 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-10T18:56:28.575 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-10T18:56:28.575 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-10T18:56:28.575 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-10T18:56:28.575 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-10T18:56:28.575 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-10T18:56:28.575 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-10T18:56:28.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-10T18:56:28.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [osd] 2024-09-10T18:56:28.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-10T18:56:28.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-10T18:56:28.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.920+0000 7f9079a1b640 1 Processor -- start 2024-09-10T18:56:28.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.920+0000 7f9079a1b640 1 -- start start 2024-09-10T18:56:28.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.920+0000 7f9079a1b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90740ff440 0x7f90740ff850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:28.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.920+0000 7f9079a1b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90740ffd90 con 0x7f90740ff440 2024-09-10T18:56:28.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.920+0000 7f9072ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90740ff440 0x7f90740ff850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:28.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.920+0000 7f9072ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90740ff440 0x7f90740ff850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45958/0 (socket says 172.21.15.119:45958) 2024-09-10T18:56:28.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.920+0000 7f9072ffd640 1 -- 172.21.15.119:0/3405172335 learned_addr learned my addr 172.21.15.119:0/3405172335 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:28.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.920+0000 7f9072ffd640 1 -- 172.21.15.119:0/3405172335 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90740ffed0 con 0x7f90740ff440 2024-09-10T18:56:28.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.921+0000 7f9072ffd640 1 --2- 172.21.15.119:0/3405172335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90740ff440 0x7f90740ff850 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9064009840 tx=0x7f90640091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82c84549495ab3cb server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:28.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.921+0000 7f9071ffb640 1 -- 172.21.15.119:0/3405172335 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f906402add0 con 0x7f90740ff440 2024-09-10T18:56:28.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.921+0000 7f9071ffb640 1 -- 172.21.15.119:0/3405172335 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9064032440 con 0x7f90740ff440 2024-09-10T18:56:28.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.921+0000 7f9071ffb640 1 -- 172.21.15.119:0/3405172335 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9064032750 con 0x7f90740ff440 2024-09-10T18:56:28.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.922+0000 7f9079a1b640 1 -- 172.21.15.119:0/3405172335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90740ff440 msgr2=0x7f90740ff850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:28.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.922+0000 7f9079a1b640 1 --2- 172.21.15.119:0/3405172335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90740ff440 0x7f90740ff850 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9064009840 tx=0x7f90640091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:28.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.922+0000 7f9079a1b640 1 -- 172.21.15.119:0/3405172335 shutdown_connections 2024-09-10T18:56:28.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.922+0000 7f9079a1b640 1 --2- 172.21.15.119:0/3405172335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90740ff440 0x7f90740ff850 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:28.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.922+0000 7f9079a1b640 1 -- 172.21.15.119:0/3405172335 >> 172.21.15.119:0/3405172335 conn(0x7f90740fabc0 msgr2=0x7f90740fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:28.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.922+0000 7f9079a1b640 1 -- 172.21.15.119:0/3405172335 shutdown_connections 2024-09-10T18:56:28.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.922+0000 7f9079a1b640 1 -- 172.21.15.119:0/3405172335 wait complete. 2024-09-10T18:56:28.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.922+0000 7f9079a1b640 1 Processor -- start 2024-09-10T18:56:28.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.923+0000 7f9079a1b640 1 -- start start 2024-09-10T18:56:28.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.923+0000 7f9079a1b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90740ff440 0x7f907410d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:28.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.923+0000 7f9079a1b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907410df30 con 0x7f90740ff440 2024-09-10T18:56:28.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.923+0000 7f9072ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90740ff440 0x7f907410d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:28.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.923+0000 7f9072ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90740ff440 0x7f907410d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45960/0 (socket says 172.21.15.119:45960) 2024-09-10T18:56:28.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.923+0000 7f9072ffd640 1 -- 172.21.15.119:0/2159574148 learned_addr learned my addr 172.21.15.119:0/2159574148 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:28.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.923+0000 7f9072ffd640 1 -- 172.21.15.119:0/2159574148 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9064009450 con 0x7f90740ff440 2024-09-10T18:56:28.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.923+0000 7f9072ffd640 1 --2- 172.21.15.119:0/2159574148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90740ff440 0x7f907410d9f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f906402a700 tx=0x7f9064032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:28.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.923+0000 7f905b7fe640 1 -- 172.21.15.119:0/2159574148 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9064031a80 con 0x7f90740ff440 2024-09-10T18:56:28.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.923+0000 7f905b7fe640 1 -- 172.21.15.119:0/2159574148 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9064031be0 con 0x7f90740ff440 2024-09-10T18:56:28.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.923+0000 7f9079a1b640 1 -- 172.21.15.119:0/2159574148 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907410e130 con 0x7f90740ff440 2024-09-10T18:56:28.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.923+0000 7f905b7fe640 1 -- 172.21.15.119:0/2159574148 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f906403b400 con 0x7f90740ff440 2024-09-10T18:56:28.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.923+0000 7f9079a1b640 1 -- 172.21.15.119:0/2159574148 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907410e540 con 0x7f90740ff440 2024-09-10T18:56:28.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.924+0000 7f905b7fe640 1 -- 172.21.15.119:0/2159574148 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f9064039040 con 0x7f90740ff440 2024-09-10T18:56:28.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.924+0000 7f905b7fe640 1 -- 172.21.15.119:0/2159574148 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9064041030 con 0x7f90740ff440 2024-09-10T18:56:28.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.924+0000 7f9079a1b640 1 -- 172.21.15.119:0/2159574148 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907404b720 con 0x7f90740ff440 2024-09-10T18:56:28.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.926+0000 7f905b7fe640 1 -- 172.21.15.119:0/2159574148 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f9064037070 con 0x7f90740ff440 2024-09-10T18:56:28.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.981+0000 7f9079a1b640 1 -- 172.21.15.119:0/2159574148 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f90741007d0 con 0x7f90740ff440 2024-09-10T18:56:28.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.984+0000 7f905b7fe640 1 -- 172.21.15.119:0/2159574148 <== mon.0 v2:172.21.15.119:3300/0 7 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f90640467d0 con 0x7f90740ff440 2024-09-10T18:56:28.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.985+0000 7f905b7fe640 1 -- 172.21.15.119:0/2159574148 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+376 (secure 0 0 0) 0x7f9064055020 con 0x7f90740ff440 2024-09-10T18:56:28.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.985+0000 7f9079a1b640 1 -- 172.21.15.119:0/2159574148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90740ff440 msgr2=0x7f907410d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:28.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.985+0000 7f9079a1b640 1 --2- 172.21.15.119:0/2159574148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90740ff440 0x7f907410d9f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f906402a700 tx=0x7f9064032000 comp rx=0 tx=0).stop 2024-09-10T18:56:28.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.985+0000 7f9079a1b640 1 -- 172.21.15.119:0/2159574148 shutdown_connections 2024-09-10T18:56:28.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.985+0000 7f9079a1b640 1 --2- 172.21.15.119:0/2159574148 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90740ff440 0x7f907410d9f0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:28.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.985+0000 7f9079a1b640 1 -- 172.21.15.119:0/2159574148 >> 172.21.15.119:0/2159574148 conn(0x7f90740fabc0 msgr2=0x7f9074104270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:28.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.985+0000 7f9079a1b640 1 -- 172.21.15.119:0/2159574148 shutdown_connections 2024-09-10T18:56:28.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:27.985+0000 7f9079a1b640 1 -- 172.21.15.119:0/2159574148 wait complete. 2024-09-10T18:56:28.579 INFO:teuthology.orchestra.run.smithi119.stdout:Generating new minimal ceph.conf... 2024-09-10T18:56:29.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:29 smithi119 ceph-mon[29107]: from='client.? 172.21.15.119:0/93656806' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:56:29.671 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.993+0000 7f1ddd2ad640 1 Processor -- start 2024-09-10T18:56:29.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.993+0000 7f1ddd2ad640 1 -- start start 2024-09-10T18:56:29.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.993+0000 7f1ddd2ad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd81006c0 0x7f1dd8100ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:29.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.993+0000 7f1ddd2ad640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd8101010 con 0x7f1dd81006c0 2024-09-10T18:56:29.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.993+0000 7f1dd6ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd81006c0 0x7f1dd8100ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:29.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.993+0000 7f1dd6ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd81006c0 0x7f1dd8100ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45964/0 (socket says 172.21.15.119:45964) 2024-09-10T18:56:29.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.993+0000 7f1dd6ffd640 1 -- 172.21.15.119:0/1986027870 learned_addr learned my addr 172.21.15.119:0/1986027870 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:29.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.993+0000 7f1dd6ffd640 1 -- 172.21.15.119:0/1986027870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd8101150 con 0x7f1dd81006c0 2024-09-10T18:56:29.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.994+0000 7f1dd6ffd640 1 --2- 172.21.15.119:0/1986027870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd81006c0 0x7f1dd8100ad0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8009840 tx=0x7f1dc80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f151f306b1105d0c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:29.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.994+0000 7f1dd5ffb640 1 -- 172.21.15.119:0/1986027870 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1dc802ae60 con 0x7f1dd81006c0 2024-09-10T18:56:29.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.994+0000 7f1dd5ffb640 1 -- 172.21.15.119:0/1986027870 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f1dc8032890 con 0x7f1dd81006c0 2024-09-10T18:56:29.672 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.994+0000 7f1dd5ffb640 1 -- 172.21.15.119:0/1986027870 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1dc80317a0 con 0x7f1dd81006c0 2024-09-10T18:56:29.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.995+0000 7f1ddd2ad640 1 -- 172.21.15.119:0/1986027870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd81006c0 msgr2=0x7f1dd8100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:29.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.995+0000 7f1ddd2ad640 1 --2- 172.21.15.119:0/1986027870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd81006c0 0x7f1dd8100ad0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1dc8009840 tx=0x7f1dc80091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:29.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.995+0000 7f1ddd2ad640 1 -- 172.21.15.119:0/1986027870 shutdown_connections 2024-09-10T18:56:29.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.995+0000 7f1ddd2ad640 1 --2- 172.21.15.119:0/1986027870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd81006c0 0x7f1dd8100ad0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:29.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.995+0000 7f1ddd2ad640 1 -- 172.21.15.119:0/1986027870 >> 172.21.15.119:0/1986027870 conn(0x7f1dd80fbea0 msgr2=0x7f1dd80fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:29.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.995+0000 7f1ddd2ad640 1 -- 172.21.15.119:0/1986027870 shutdown_connections 2024-09-10T18:56:29.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.995+0000 7f1ddd2ad640 1 -- 172.21.15.119:0/1986027870 wait complete. 2024-09-10T18:56:29.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.995+0000 7f1ddd2ad640 1 Processor -- start 2024-09-10T18:56:29.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.995+0000 7f1ddd2ad640 1 -- start start 2024-09-10T18:56:29.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.996+0000 7f1ddd2ad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd81006c0 0x7f1dd810f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:29.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.996+0000 7f1ddd2ad640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd810f7d0 con 0x7f1dd81006c0 2024-09-10T18:56:29.673 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.996+0000 7f1dd6ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd81006c0 0x7f1dd810f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:29.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.996+0000 7f1dd6ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd81006c0 0x7f1dd810f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45968/0 (socket says 172.21.15.119:45968) 2024-09-10T18:56:29.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.996+0000 7f1dd6ffd640 1 -- 172.21.15.119:0/93656806 learned_addr learned my addr 172.21.15.119:0/93656806 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:29.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.996+0000 7f1dd6ffd640 1 -- 172.21.15.119:0/93656806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc8009450 con 0x7f1dd81006c0 2024-09-10T18:56:29.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.996+0000 7f1dd6ffd640 1 --2- 172.21.15.119:0/93656806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd81006c0 0x7f1dd810f290 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1dc80313e0 tx=0x7f1dc80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:29.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.996+0000 7f1dbf7fe640 1 -- 172.21.15.119:0/93656806 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1dc8031be0 con 0x7f1dd81006c0 2024-09-10T18:56:29.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.996+0000 7f1dbf7fe640 1 -- 172.21.15.119:0/93656806 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f1dc8031d40 con 0x7f1dd81006c0 2024-09-10T18:56:29.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.996+0000 7f1ddd2ad640 1 -- 172.21.15.119:0/93656806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd810f9d0 con 0x7f1dd81006c0 2024-09-10T18:56:29.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.996+0000 7f1dbf7fe640 1 -- 172.21.15.119:0/93656806 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1dc803bda0 con 0x7f1dd81006c0 2024-09-10T18:56:29.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.996+0000 7f1ddd2ad640 1 -- 172.21.15.119:0/93656806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd810fe40 con 0x7f1dd81006c0 2024-09-10T18:56:29.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.996+0000 7f1dbf7fe640 1 -- 172.21.15.119:0/93656806 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f1dc8039040 con 0x7f1dd81006c0 2024-09-10T18:56:29.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.996+0000 7f1dbf7fe640 1 -- 172.21.15.119:0/93656806 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1dc803a420 con 0x7f1dd81006c0 2024-09-10T18:56:29.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.997+0000 7f1ddd2ad640 1 -- 172.21.15.119:0/93656806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd804b720 con 0x7f1dd81006c0 2024-09-10T18:56:29.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:28.999+0000 7f1dbf7fe640 1 -- 172.21.15.119:0/93656806 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f1dd810fe40 con 0x7f1dd81006c0 2024-09-10T18:56:29.674 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:29.054+0000 7f1ddd2ad640 1 -- 172.21.15.119:0/93656806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f1dd8110600 con 0x7f1dd81006c0 2024-09-10T18:56:29.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:29.054+0000 7f1dbf7fe640 1 -- 172.21.15.119:0/93656806 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f1dc8049060 con 0x7f1dd81006c0 2024-09-10T18:56:29.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:29.055+0000 7f1ddd2ad640 1 -- 172.21.15.119:0/93656806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd81006c0 msgr2=0x7f1dd810f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:29.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:29.055+0000 7f1ddd2ad640 1 --2- 172.21.15.119:0/93656806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd81006c0 0x7f1dd810f290 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1dc80313e0 tx=0x7f1dc80091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:29.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:29.055+0000 7f1ddd2ad640 1 -- 172.21.15.119:0/93656806 shutdown_connections 2024-09-10T18:56:29.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:29.055+0000 7f1ddd2ad640 1 --2- 172.21.15.119:0/93656806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1dd81006c0 0x7f1dd810f290 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:29.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:29.055+0000 7f1ddd2ad640 1 -- 172.21.15.119:0/93656806 >> 172.21.15.119:0/93656806 conn(0x7f1dd80fbea0 msgr2=0x7f1dd80fc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:29.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:29.055+0000 7f1ddd2ad640 1 -- 172.21.15.119:0/93656806 shutdown_connections 2024-09-10T18:56:29.675 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:29.055+0000 7f1ddd2ad640 1 -- 172.21.15.119:0/93656806 wait complete. 2024-09-10T18:56:29.675 INFO:teuthology.orchestra.run.smithi119.stdout:Restarting the monitor... 2024-09-10T18:56:30.014 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:29 smithi119 systemd[1]: Stopping Ceph mon.a for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T18:56:30.014 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:29 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a[29084]: 2024-09-10T18:56:29.910+0000 7f37c6739640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T18:56:30.014 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:29 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a[29084]: 2024-09-10T18:56:29.910+0000 7f37c6739640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-10T18:56:30.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:30 smithi119 podman[29423]: 2024-09-10 18:56:30.012975739 +0000 UTC m=+0.183111904 container died edd7a8c744e9ed1922fb0f3952e9776a29bf110b276be69c6471c90030999618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True) 2024-09-10T18:56:30.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:30 smithi119 podman[29423]: 2024-09-10 18:56:30.375548567 +0000 UTC m=+0.545684775 container cleanup edd7a8c744e9ed1922fb0f3952e9776a29bf110b276be69c6471c90030999618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:56:30.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:30 smithi119 bash[29423]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a 2024-09-10T18:56:30.994 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:30 smithi119 podman[29443]: 2024-09-10 18:56:30.661156798 +0000 UTC m=+0.644488209 container remove edd7a8c744e9ed1922fb0f3952e9776a29bf110b276be69c6471c90030999618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-09-10T18:56:30.994 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:30 smithi119 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-10T18:56:30.994 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:30 smithi119 systemd[1]: Stopped Ceph mon.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T18:56:30.994 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:30 smithi119 systemd[1]: Starting Ceph mon.a for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T18:56:31.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:30 smithi119 podman[29540]: 2024-09-10 18:56:30.895735755 +0000 UTC m=+0.015397823 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:56:31.247 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 podman[29540]: 2024-09-10 18:56:31.010400036 +0000 UTC m=+0.130062089 container create f867ead4cf34929f3d407e5b032218276626003d1e36c4f22d3c814200cb81cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-10T18:56:31.247 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 podman[29540]: 2024-09-10 18:56:31.24615208 +0000 UTC m=+0.365814140 container init f867ead4cf34929f3d407e5b032218276626003d1e36c4f22d3c814200cb81cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T18:56:31.318 INFO:teuthology.orchestra.run.smithi119.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-10T18:56:31.512 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 podman[29540]: 2024-09-10 18:56:31.250951082 +0000 UTC m=+0.370613136 container start f867ead4cf34929f3d407e5b032218276626003d1e36c4f22d3c814200cb81cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-10T18:56:31.512 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: set uid:gid to 167:167 (ceph:ceph) 2024-09-10T18:56:31.512 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable), process ceph-mon, pid 7 2024-09-10T18:56:31.512 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: pidfile_write: ignore empty --pid-file 2024-09-10T18:56:31.512 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: load: jerasure load: lrc 2024-09-10T18:56:31.512 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: RocksDB version: 6.15.5 2024-09-10T18:56:31.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-10T18:56:31.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Compile date Sep 9 2024 2024-09-10T18:56:31.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: DB SUMMARY 2024-09-10T18:56:31.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: DB Session ID: SSVXYGOGKVXL4XTAP12L 2024-09-10T18:56:31.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: CURRENT file: CURRENT 2024-09-10T18:56:31.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: IDENTITY file: IDENTITY 2024-09-10T18:56:31.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-10T18:56:31.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-10T18:56:31.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 85954 ; 2024-09-10T18:56:31.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.error_if_exists: 0 2024-09-10T18:56:31.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.create_if_missing: 0 2024-09-10T18:56:31.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.paranoid_checks: 1 2024-09-10T18:56:31.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-10T18:56:31.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.env: 0x5643ddf8a440 2024-09-10T18:56:31.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.fs: Posix File System 2024-09-10T18:56:31.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.info_log: 0x5643dfcb8ec0 2024-09-10T18:56:31.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_file_opening_threads: 16 2024-09-10T18:56:31.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.statistics: (nil) 2024-09-10T18:56:31.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.use_fsync: 0 2024-09-10T18:56:31.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_log_file_size: 0 2024-09-10T18:56:31.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-10T18:56:31.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-10T18:56:31.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.keep_log_file_num: 1000 2024-09-10T18:56:31.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.recycle_log_file_num: 0 2024-09-10T18:56:31.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.allow_fallocate: 1 2024-09-10T18:56:31.514 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.allow_mmap_reads: 0 2024-09-10T18:56:31.515 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.allow_mmap_writes: 0 2024-09-10T18:56:31.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.use_direct_reads: 0 2024-09-10T18:56:31.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-10T18:56:31.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.create_missing_column_families: 0 2024-09-10T18:56:31.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.db_log_dir: 2024-09-10T18:56:31.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-10T18:56:31.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-10T18:56:31.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-10T18:56:31.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-10T18:56:31.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-10T18:56:31.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-10T18:56:31.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-10T18:56:31.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.advise_random_on_open: 1 2024-09-10T18:56:31.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.db_write_buffer_size: 0 2024-09-10T18:56:31.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.write_buffer_manager: 0x5643dfd76780 2024-09-10T18:56:31.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-10T18:56:31.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-10T18:56:31.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-10T18:56:31.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-10T18:56:31.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.rate_limiter: (nil) 2024-09-10T18:56:31.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-10T18:56:31.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.wal_recovery_mode: 2 2024-09-10T18:56:31.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.enable_thread_tracking: 0 2024-09-10T18:56:31.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.enable_pipelined_write: 0 2024-09-10T18:56:31.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.unordered_write: 0 2024-09-10T18:56:31.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-10T18:56:31.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-10T18:56:31.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-10T18:56:31.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-10T18:56:31.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.row_cache: None 2024-09-10T18:56:31.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.wal_filter: None 2024-09-10T18:56:31.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-10T18:56:31.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.allow_ingest_behind: 0 2024-09-10T18:56:31.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.preserve_deletes: 0 2024-09-10T18:56:31.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.two_write_queues: 0 2024-09-10T18:56:31.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.manual_wal_flush: 0 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.atomic_flush: 0 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.log_readahead_size: 0 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.best_efforts_recovery: 0 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.allow_data_in_errors: 0 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.db_host_id: __hostname__ 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_background_jobs: 2 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_background_compactions: -1 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_subcompactions: 1 2024-09-10T18:56:31.519 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-10T18:56:31.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-10T18:56:31.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-10T18:56:31.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_total_wal_size: 0 2024-09-10T18:56:31.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-10T18:56:31.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-10T18:56:31.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-10T18:56:31.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-10T18:56:31.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_open_files: -1 2024-09-10T18:56:31.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.bytes_per_sync: 0 2024-09-10T18:56:31.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-10T18:56:31.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-10T18:56:31.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compaction_readahead_size: 0 2024-09-10T18:56:31.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_background_flushes: -1 2024-09-10T18:56:31.520 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Compression algorithms supported: 2024-09-10T18:56:31.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: kZSTD supported: 0 2024-09-10T18:56:31.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: kXpressCompression supported: 0 2024-09-10T18:56:31.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: kBZip2Compression supported: 0 2024-09-10T18:56:31.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-10T18:56:31.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: kLZ4Compression supported: 1 2024-09-10T18:56:31.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: kZlibCompression supported: 1 2024-09-10T18:56:31.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: kLZ4HCCompression supported: 1 2024-09-10T18:56:31.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: kSnappyCompression supported: 1 2024-09-10T18:56:31.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-10T18:56:31.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-09-10T18:56:31.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-10T18:56:31.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-10T18:56:31.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.merge_operator: 2024-09-10T18:56:31.521 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compaction_filter: None 2024-09-10T18:56:31.522 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compaction_filter_factory: None 2024-09-10T18:56:31.522 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.sst_partitioner_factory: None 2024-09-10T18:56:31.522 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-10T18:56:31.522 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-10T18:56:31.522 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5643dfc5fa38) 2024-09-10T18:56:31.522 INFO:journalctl@ceph.mon.a.smithi119.stdout: cache_index_and_filter_blocks: 1 2024-09-10T18:56:31.522 INFO:journalctl@ceph.mon.a.smithi119.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-10T18:56:31.522 INFO:journalctl@ceph.mon.a.smithi119.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-10T18:56:31.522 INFO:journalctl@ceph.mon.a.smithi119.stdout: pin_top_level_index_and_filter: 1 2024-09-10T18:56:31.522 INFO:journalctl@ceph.mon.a.smithi119.stdout: index_type: 0 2024-09-10T18:56:31.522 INFO:journalctl@ceph.mon.a.smithi119.stdout: data_block_index_type: 0 2024-09-10T18:56:31.522 INFO:journalctl@ceph.mon.a.smithi119.stdout: index_shortening: 1 2024-09-10T18:56:31.523 INFO:journalctl@ceph.mon.a.smithi119.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-10T18:56:31.523 INFO:journalctl@ceph.mon.a.smithi119.stdout: hash_index_allow_collision: 1 2024-09-10T18:56:31.523 INFO:journalctl@ceph.mon.a.smithi119.stdout: checksum: 1 2024-09-10T18:56:31.523 INFO:journalctl@ceph.mon.a.smithi119.stdout: no_block_cache: 0 2024-09-10T18:56:31.523 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache: 0x5643dfc951f0 2024-09-10T18:56:31.523 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache_name: BinnedLRUCache 2024-09-10T18:56:31.523 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache_options: 2024-09-10T18:56:31.523 INFO:journalctl@ceph.mon.a.smithi119.stdout: capacity : 536870912 2024-09-10T18:56:31.523 INFO:journalctl@ceph.mon.a.smithi119.stdout: num_shard_bits : 4 2024-09-10T18:56:31.523 INFO:journalctl@ceph.mon.a.smithi119.stdout: strict_capacity_limit : 0 2024-09-10T18:56:31.523 INFO:journalctl@ceph.mon.a.smithi119.stdout: high_pri_pool_ratio: 0.000 2024-09-10T18:56:31.523 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_cache_compressed: (nil) 2024-09-10T18:56:31.523 INFO:journalctl@ceph.mon.a.smithi119.stdout: persistent_cache: (nil) 2024-09-10T18:56:31.523 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_size: 4096 2024-09-10T18:56:31.524 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_size_deviation: 10 2024-09-10T18:56:31.524 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_restart_interval: 16 2024-09-10T18:56:31.524 INFO:journalctl@ceph.mon.a.smithi119.stdout: index_block_restart_interval: 1 2024-09-10T18:56:31.524 INFO:journalctl@ceph.mon.a.smithi119.stdout: metadata_block_size: 4096 2024-09-10T18:56:31.524 INFO:journalctl@ceph.mon.a.smithi119.stdout: partition_filters: 0 2024-09-10T18:56:31.524 INFO:journalctl@ceph.mon.a.smithi119.stdout: use_delta_encoding: 1 2024-09-10T18:56:31.524 INFO:journalctl@ceph.mon.a.smithi119.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-10T18:56:31.524 INFO:journalctl@ceph.mon.a.smithi119.stdout: whole_key_filtering: 1 2024-09-10T18:56:31.524 INFO:journalctl@ceph.mon.a.smithi119.stdout: verify_compression: 0 2024-09-10T18:56:31.524 INFO:journalctl@ceph.mon.a.smithi119.stdout: read_amp_bytes_per_bit: 0 2024-09-10T18:56:31.524 INFO:journalctl@ceph.mon.a.smithi119.stdout: format_version: 4 2024-09-10T18:56:31.524 INFO:journalctl@ceph.mon.a.smithi119.stdout: enable_index_compression: 1 2024-09-10T18:56:31.524 INFO:journalctl@ceph.mon.a.smithi119.stdout: block_align: 0 2024-09-10T18:56:31.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.write_buffer_size: 33554432 2024-09-10T18:56:31.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_write_buffer_number: 2 2024-09-10T18:56:31.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compression: NoCompression 2024-09-10T18:56:31.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.bottommost_compression: Disabled 2024-09-10T18:56:31.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.prefix_extractor: nullptr 2024-09-10T18:56:31.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-10T18:56:31.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.num_levels: 7 2024-09-10T18:56:31.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-10T18:56:31.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-10T18:56:31.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-10T18:56:31.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-10T18:56:31.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-10T18:56:31.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-10T18:56:31.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-10T18:56:31.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:56:31.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-10T18:56:31.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-10T18:56:31.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-10T18:56:31.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compression_opts.level: 32767 2024-09-10T18:56:31.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compression_opts.strategy: 0 2024-09-10T18:56:31.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-10T18:56:31.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:56:31.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-10T18:56:31.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compression_opts.enabled: false 2024-09-10T18:56:31.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-10T18:56:31.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-10T18:56:31.526 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-10T18:56:31.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.target_file_size_base: 67108864 2024-09-10T18:56:31.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-10T18:56:31.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-10T18:56:31.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-10T18:56:31.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-10T18:56:31.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-10T18:56:31.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-10T18:56:31.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-10T18:56:31.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-10T18:56:31.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-10T18:56:31.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-10T18:56:31.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-10T18:56:31.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-10T18:56:31.527 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-10T18:56:31.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.arena_block_size: 4194304 2024-09-10T18:56:31.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-10T18:56:31.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-10T18:56:31.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-10T18:56:31.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.disable_auto_compactions: 0 2024-09-10T18:56:31.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-10T18:56:31.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-10T18:56:31.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-10T18:56:31.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-10T18:56:31.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-10T18:56:31.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-10T18:56:31.528 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-10T18:56:31.529 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-10T18:56:31.529 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-10T18:56:31.529 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-10T18:56:31.529 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-10T18:56:31.529 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.inplace_update_support: 0 2024-09-10T18:56:31.529 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-10T18:56:31.529 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-10T18:56:31.529 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-10T18:56:31.529 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-10T18:56:31.529 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.bloom_locality: 0 2024-09-10T18:56:31.529 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.max_successive_merges: 0 2024-09-10T18:56:31.529 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-10T18:56:31.529 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.paranoid_file_checks: 0 2024-09-10T18:56:31.530 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.force_consistency_checks: 1 2024-09-10T18:56:31.530 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.report_bg_io_stats: 0 2024-09-10T18:56:31.530 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.ttl: 2592000 2024-09-10T18:56:31.530 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-10T18:56:31.530 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.enable_blob_files: false 2024-09-10T18:56:31.530 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.min_blob_size: 0 2024-09-10T18:56:31.530 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.blob_file_size: 268435456 2024-09-10T18:56:31.530 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-10T18:56:31.530 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-10T18:56:31.530 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-10T18:56:31.530 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-10T18:56:31.530 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-09-10T18:56:31.530 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-09-10T18:56:31.530 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725994591274553, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-10T18:56:31.531 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-09-10T18:56:31.531 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-10T18:56:31.531 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725994591275287, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 82256, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 80502, "index_size": 216, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10546, "raw_average_key_size": 47, "raw_value_size": 74606, "raw_average_value_size": 337, "num_data_blocks": 10, "num_entries": 221, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1725994591, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "5eae89db-e3fd-4c24-86f9-f157d8ec13a0", "db_session_id": "SSVXYGOGKVXL4XTAP12L"}} 2024-09-10T18:56:31.531 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-09-10T18:56:31.531 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725994591275653, "job": 1, "event": "recovery_finished"} 2024-09-10T18:56:31.531 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-10T18:56:31.531 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5643dfcc8700 2024-09-10T18:56:31.531 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: DB pointer 0x5643dfc98000 2024-09-10T18:56:31.531 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-10T18:56:31.531 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-10T18:56:31.531 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** DB Stats ** 2024-09-10T18:56:31.531 INFO:journalctl@ceph.mon.a.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:56:31.531 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-10T18:56:31.532 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-10T18:56:31.532 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:56:31.532 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-10T18:56:31.532 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-10T18:56:31.532 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:56:31.532 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-10T18:56:31.532 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-10T18:56:31.532 INFO:journalctl@ceph.mon.a.smithi119.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:56:31.532 INFO:journalctl@ceph.mon.a.smithi119.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:56:31.532 INFO:journalctl@ceph.mon.a.smithi119.stdout: L0 2/0 82.11 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.8 0.00 0.00 1 0.000 0 0 2024-09-10T18:56:31.532 INFO:journalctl@ceph.mon.a.smithi119.stdout: Sum 2/0 82.11 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.8 0.00 0.00 1 0.000 0 0 2024-09-10T18:56:31.532 INFO:journalctl@ceph.mon.a.smithi119.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.8 0.00 0.00 1 0.000 0 0 2024-09-10T18:56:31.532 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-10T18:56:31.533 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-10T18:56:31.533 INFO:journalctl@ceph.mon.a.smithi119.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:56:31.533 INFO:journalctl@ceph.mon.a.smithi119.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:56:31.533 INFO:journalctl@ceph.mon.a.smithi119.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.8 0.00 0.00 1 0.000 0 0 2024-09-10T18:56:31.533 INFO:journalctl@ceph.mon.a.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:56:31.533 INFO:journalctl@ceph.mon.a.smithi119.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:56:31.533 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:56:31.533 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:56:31.533 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:56:31.533 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:56:31.533 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative compaction: 0.00 GB write, 25.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:56:31.533 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval compaction: 0.00 GB write, 25.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:56:31.533 INFO:journalctl@ceph.mon.a.smithi119.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:56:31.534 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-10T18:56:31.534 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:56:31.534 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-10T18:56:31.534 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-10T18:56:31.534 INFO:journalctl@ceph.mon.a.smithi119.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:56:31.534 INFO:journalctl@ceph.mon.a.smithi119.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:56:31.534 INFO:journalctl@ceph.mon.a.smithi119.stdout: L0 2/0 82.11 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.8 0.00 0.00 1 0.000 0 0 2024-09-10T18:56:31.534 INFO:journalctl@ceph.mon.a.smithi119.stdout: Sum 2/0 82.11 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.8 0.00 0.00 1 0.000 0 0 2024-09-10T18:56:31.534 INFO:journalctl@ceph.mon.a.smithi119.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-10T18:56:31.534 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-10T18:56:31.534 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-10T18:56:31.534 INFO:journalctl@ceph.mon.a.smithi119.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:56:31.534 INFO:journalctl@ceph.mon.a.smithi119.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:56:31.534 INFO:journalctl@ceph.mon.a.smithi119.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.8 0.00 0.00 1 0.000 0 0 2024-09-10T18:56:31.535 INFO:journalctl@ceph.mon.a.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:56:31.535 INFO:journalctl@ceph.mon.a.smithi119.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:56:31.535 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:56:31.535 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:56:31.535 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:56:31.535 INFO:journalctl@ceph.mon.a.smithi119.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:56:31.535 INFO:journalctl@ceph.mon.a.smithi119.stdout: Cumulative compaction: 0.00 GB write, 25.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:56:31.535 INFO:journalctl@ceph.mon.a.smithi119.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:56:31.535 INFO:journalctl@ceph.mon.a.smithi119.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:56:31.535 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-10T18:56:31.535 INFO:journalctl@ceph.mon.a.smithi119.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:56:31.535 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: starting mon.a rank 0 at public addrs [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] at bind addrs [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:56:31.535 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: mon.a@-1(???) e1 preinit fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:56:31.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: mon.a@-1(???).mds e1 new map 2024-09-10T18:56:31.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: mon.a@-1(???).mds e1 print_map 2024-09-10T18:56:31.536 INFO:journalctl@ceph.mon.a.smithi119.stdout: e1 2024-09-10T18:56:31.536 INFO:journalctl@ceph.mon.a.smithi119.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-10T18:56:31.536 INFO:journalctl@ceph.mon.a.smithi119.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-10T18:56:31.536 INFO:journalctl@ceph.mon.a.smithi119.stdout: legacy client fscid: -1 2024-09-10T18:56:31.536 INFO:journalctl@ceph.mon.a.smithi119.stdout: 2024-09-10T18:56:31.536 INFO:journalctl@ceph.mon.a.smithi119.stdout: No filesystems configured 2024-09-10T18:56:31.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-10T18:56:31.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:56:31.536 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:56:31.537 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:56:31.537 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-10T18:56:31.537 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 bash[29540]: f867ead4cf34929f3d407e5b032218276626003d1e36c4f22d3c814200cb81cb 2024-09-10T18:56:31.537 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 systemd[1]: Started Ceph mon.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T18:56:31.537 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-10T18:56:31.537 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: monmap e1: 1 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:56:31.537 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: fsmap 2024-09-10T18:56:31.537 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: osdmap e1: 0 total, 0 up, 0 in 2024-09-10T18:56:31.537 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:31 smithi119 ceph-mon[29577]: mgrmap e1: no daemons active 2024-09-10T18:56:32.352 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.735+0000 7fd29e5e4640 1 Processor -- start 2024-09-10T18:56:32.353 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.736+0000 7fd29e5e4640 1 -- start start 2024-09-10T18:56:32.353 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.736+0000 7fd29e5e4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2980ff440 0x7fd2980ff850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:32.353 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.736+0000 7fd29e5e4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2980ffd90 con 0x7fd2980ff440 2024-09-10T18:56:32.353 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.736+0000 7fd297fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2980ff440 0x7fd2980ff850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:32.353 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.736+0000 7fd297fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2980ff440 0x7fd2980ff850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45978/0 (socket says 172.21.15.119:45978) 2024-09-10T18:56:32.353 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.736+0000 7fd297fff640 1 -- 172.21.15.119:0/2403101176 learned_addr learned my addr 172.21.15.119:0/2403101176 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:32.353 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.736+0000 7fd297fff640 1 -- 172.21.15.119:0/2403101176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2980ffed0 con 0x7fd2980ff440 2024-09-10T18:56:32.353 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.738+0000 7fd297fff640 1 --2- 172.21.15.119:0/2403101176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2980ff440 0x7fd2980ff850 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd288009840 tx=0x7fd2880091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561a29f8c70bebe5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:32.353 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.738+0000 7fd296ffd640 1 -- 172.21.15.119:0/2403101176 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd28802add0 con 0x7fd2980ff440 2024-09-10T18:56:32.353 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.738+0000 7fd296ffd640 1 -- 172.21.15.119:0/2403101176 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7fd288032890 con 0x7fd2980ff440 2024-09-10T18:56:32.353 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.738+0000 7fd296ffd640 1 -- 172.21.15.119:0/2403101176 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd2880317a0 con 0x7fd2980ff440 2024-09-10T18:56:32.354 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.739+0000 7fd29e5e4640 1 -- 172.21.15.119:0/2403101176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2980ff440 msgr2=0x7fd2980ff850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:32.354 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.739+0000 7fd29e5e4640 1 --2- 172.21.15.119:0/2403101176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2980ff440 0x7fd2980ff850 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fd288009840 tx=0x7fd2880091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:32.354 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.739+0000 7fd29e5e4640 1 -- 172.21.15.119:0/2403101176 shutdown_connections 2024-09-10T18:56:32.354 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.739+0000 7fd29e5e4640 1 --2- 172.21.15.119:0/2403101176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2980ff440 0x7fd2980ff850 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:32.354 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.739+0000 7fd29e5e4640 1 -- 172.21.15.119:0/2403101176 >> 172.21.15.119:0/2403101176 conn(0x7fd2980fabc0 msgr2=0x7fd2980fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:32.354 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.739+0000 7fd29e5e4640 1 -- 172.21.15.119:0/2403101176 shutdown_connections 2024-09-10T18:56:32.354 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.739+0000 7fd29e5e4640 1 -- 172.21.15.119:0/2403101176 wait complete. 2024-09-10T18:56:32.354 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.739+0000 7fd29e5e4640 1 Processor -- start 2024-09-10T18:56:32.354 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.740+0000 7fd29e5e4640 1 -- start start 2024-09-10T18:56:32.354 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.740+0000 7fd29e5e4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2980ff440 0x7fd29810f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:32.354 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.740+0000 7fd29e5e4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29810f7d0 con 0x7fd2980ff440 2024-09-10T18:56:32.354 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.740+0000 7fd297fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2980ff440 0x7fd29810f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:32.355 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.740+0000 7fd297fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2980ff440 0x7fd29810f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45982/0 (socket says 172.21.15.119:45982) 2024-09-10T18:56:32.355 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.740+0000 7fd297fff640 1 -- 172.21.15.119:0/1820062417 learned_addr learned my addr 172.21.15.119:0/1820062417 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:32.355 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.740+0000 7fd297fff640 1 -- 172.21.15.119:0/1820062417 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd288009450 con 0x7fd2980ff440 2024-09-10T18:56:32.355 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.740+0000 7fd297fff640 1 --2- 172.21.15.119:0/1820062417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2980ff440 0x7fd29810f290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd2880314e0 tx=0x7fd2880091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:32.355 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.740+0000 7fd294ff9640 1 -- 172.21.15.119:0/1820062417 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd28803c400 con 0x7fd2980ff440 2024-09-10T18:56:32.355 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.740+0000 7fd29e5e4640 1 -- 172.21.15.119:0/1820062417 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29810f9d0 con 0x7fd2980ff440 2024-09-10T18:56:32.355 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.740+0000 7fd294ff9640 1 -- 172.21.15.119:0/1820062417 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7fd28803c9b0 con 0x7fd2980ff440 2024-09-10T18:56:32.355 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.740+0000 7fd294ff9640 1 -- 172.21.15.119:0/1820062417 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd28803a530 con 0x7fd2980ff440 2024-09-10T18:56:32.355 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.740+0000 7fd29e5e4640 1 -- 172.21.15.119:0/1820062417 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29810fe40 con 0x7fd2980ff440 2024-09-10T18:56:32.355 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.741+0000 7fd294ff9640 1 -- 172.21.15.119:0/1820062417 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fd28803aa80 con 0x7fd2980ff440 2024-09-10T18:56:32.355 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.741+0000 7fd294ff9640 1 -- 172.21.15.119:0/1820062417 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd288046c40 con 0x7fd2980ff440 2024-09-10T18:56:32.356 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.741+0000 7fd29e5e4640 1 -- 172.21.15.119:0/1820062417 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29804b720 con 0x7fd2980ff440 2024-09-10T18:56:32.356 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.743+0000 7fd294ff9640 1 -- 172.21.15.119:0/1820062417 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7fd29810fe40 con 0x7fd2980ff440 2024-09-10T18:56:32.356 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.798+0000 7fd29e5e4640 1 -- 172.21.15.119:0/1820062417 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fd298101260 con 0x7fd2980ff440 2024-09-10T18:56:32.356 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.801+0000 7fd294ff9640 1 -- 172.21.15.119:0/1820062417 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fd288037070 con 0x7fd2980ff440 2024-09-10T18:56:32.356 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.802+0000 7fd29e5e4640 1 -- 172.21.15.119:0/1820062417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2980ff440 msgr2=0x7fd29810f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:32.356 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.802+0000 7fd29e5e4640 1 --2- 172.21.15.119:0/1820062417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2980ff440 0x7fd29810f290 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fd2880314e0 tx=0x7fd2880091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:32.356 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.802+0000 7fd29e5e4640 1 -- 172.21.15.119:0/1820062417 shutdown_connections 2024-09-10T18:56:32.356 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.802+0000 7fd29e5e4640 1 --2- 172.21.15.119:0/1820062417 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2980ff440 0x7fd29810f290 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:32.356 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.802+0000 7fd29e5e4640 1 -- 172.21.15.119:0/1820062417 >> 172.21.15.119:0/1820062417 conn(0x7fd2980fabc0 msgr2=0x7fd2980fb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:32.356 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.802+0000 7fd29e5e4640 1 -- 172.21.15.119:0/1820062417 shutdown_connections 2024-09-10T18:56:32.356 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:31.802+0000 7fd29e5e4640 1 -- 172.21.15.119:0/1820062417 wait complete. 2024-09-10T18:56:32.357 INFO:teuthology.orchestra.run.smithi119.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-10T18:56:32.357 INFO:teuthology.orchestra.run.smithi119.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-10T18:56:32.357 INFO:teuthology.orchestra.run.smithi119.stdout:Creating mgr... 2024-09-10T18:56:32.357 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-10T18:56:32.357 INFO:teuthology.orchestra.run.smithi119.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-10T18:56:32.525 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mgr.y 2024-09-10T18:56:32.525 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Failed to reset failed state of unit ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mgr.y.service: Unit ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mgr.y.service not loaded. 2024-09-10T18:56:32.680 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-3afe9804-6fa6-11ef-bcea-c7b262605968.target.wants/ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@.service. 2024-09-10T18:56:32.696 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:32 smithi119 systemd[1]: Starting Ceph mgr.y for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T18:56:33.008 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1820062417' entity='client.admin' 2024-09-10T18:56:33.008 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:32 smithi119 podman[29812]: 2024-09-10 18:56:32.809226796 +0000 UTC m=+0.016318802 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:56:33.008 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:32 smithi119 podman[29812]: 2024-09-10 18:56:32.912225302 +0000 UTC m=+0.119317305 container create 375fba9bae7df0355a5c36b86f8af75d81bdec2895c374ca4d7fae7201822fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-10T18:56:33.223 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T18:56:33.223 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout disabled 2024-09-10T18:56:33.232 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T18:56:33.232 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-09-10T18:56:33.232 INFO:teuthology.orchestra.run.smithi119.stdout:firewalld.service is not enabled 2024-09-10T18:56:33.232 INFO:teuthology.orchestra.run.smithi119.stdout:Not possible to enable service . firewalld.service is not available 2024-09-10T18:56:33.243 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T18:56:33.243 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout disabled 2024-09-10T18:56:33.252 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T18:56:33.252 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-09-10T18:56:33.252 INFO:teuthology.orchestra.run.smithi119.stdout:firewalld.service is not enabled 2024-09-10T18:56:33.252 INFO:teuthology.orchestra.run.smithi119.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-10T18:56:33.253 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mgr to start... 2024-09-10T18:56:33.253 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mgr... 2024-09-10T18:56:33.376 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:33 smithi119 podman[29812]: 2024-09-10 18:56:33.114851096 +0000 UTC m=+0.321943099 container init 375fba9bae7df0355a5c36b86f8af75d81bdec2895c374ca4d7fae7201822fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-10T18:56:33.376 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:33 smithi119 podman[29812]: 2024-09-10 18:56:33.119626519 +0000 UTC m=+0.326718522 container start 375fba9bae7df0355a5c36b86f8af75d81bdec2895c374ca4d7fae7201822fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS) 2024-09-10T18:56:33.376 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:33 smithi119 bash[29812]: 375fba9bae7df0355a5c36b86f8af75d81bdec2895c374ca4d7fae7201822fdb 2024-09-10T18:56:33.376 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:33 smithi119 systemd[1]: Started Ceph mgr.y for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T18:56:33.376 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:33.236+0000 7f07097d5ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:56:33.376 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:33.298+0000 7f07097d5ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:56:33.377 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:33.362+0000 7f07097d5ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:56:33.649 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:33.487+0000 7f07097d5ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:56:33.650 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:33.647+0000 7f07097d5ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:56:33.920 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2582398219' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-10T18:56:34.280 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:33.919+0000 7f07097d5ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:56:34.281 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:33.979+0000 7f07097d5ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:56:34.281 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:34 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:34.141+0000 7f07097d5ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:56:34.499 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-10T18:56:34.502 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:56:34.502 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsid": "3afe9804-6fa6-11ef-bcea-c7b262605968", 2024-09-10T18:56:34.502 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "health": { 2024-09-10T18:56:34.502 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-10T18:56:34.502 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-10T18:56:34.502 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-10T18:56:34.502 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:34.503 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-10T18:56:34.503 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-10T18:56:34.503 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 0 2024-09-10T18:56:34.503 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:56:34.503 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-10T18:56:34.503 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "a" 2024-09-10T18:56:34.503 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:56:34.503 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-10T18:56:34.503 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-10T18:56:34.503 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:56:34.503 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-10T18:56:34.503 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-10T18:56:34.503 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:34.504 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-10T18:56:34.504 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:56:34.504 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-10T18:56:34.504 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-10T18:56:34.504 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-10T18:56:34.511 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-10T18:56:34.511 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-10T18:56:34.511 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-10T18:56:34.511 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:34.511 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-10T18:56:34.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-10T18:56:34.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-10T18:56:34.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-10T18:56:34.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-10T18:56:34.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-10T18:56:34.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-10T18:56:34.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-10T18:56:34.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-10T18:56:34.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:34.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-10T18:56:34.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:56:34.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-10T18:56:34.512 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-10T18:56:34.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:34.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-10T18:56:34.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-10T18:56:34.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-10T18:56:34.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-10T18:56:34.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-10T18:56:34.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-10T18:56:34.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "restful" 2024-09-10T18:56:34.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:56:34.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:56:34.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:34.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-10T18:56:34.513 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:56:34.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modified": "2024-09-10T18:56:26.299768+0000", 2024-09-10T18:56:34.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:56:34.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:34.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-10T18:56:34.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:56:34.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.762+0000 7f943851e640 1 Processor -- start 2024-09-10T18:56:34.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.762+0000 7f943851e640 1 -- start start 2024-09-10T18:56:34.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.762+0000 7f943851e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94300ff440 0x7f94300ff850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:34.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.762+0000 7f943851e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94300ffd90 con 0x7f94300ff440 2024-09-10T18:56:34.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.763+0000 7f94362ba640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94300ff440 0x7f94300ff850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:34.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.763+0000 7f94362ba640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94300ff440 0x7f94300ff850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53572/0 (socket says 172.21.15.119:53572) 2024-09-10T18:56:34.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.763+0000 7f94362ba640 1 -- 172.21.15.119:0/756890807 learned_addr learned my addr 172.21.15.119:0/756890807 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:34.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.763+0000 7f94362ba640 1 -- 172.21.15.119:0/756890807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94300ffed0 con 0x7f94300ff440 2024-09-10T18:56:34.514 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.764+0000 7f94362ba640 1 --2- 172.21.15.119:0/756890807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94300ff440 0x7f94300ff850 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f942c009840 tx=0x7f942c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62abd081715df5be server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:34.515 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.764+0000 7f94352b8640 1 -- 172.21.15.119:0/756890807 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f942c02add0 con 0x7f94300ff440 2024-09-10T18:56:34.515 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.764+0000 7f94352b8640 1 -- 172.21.15.119:0/756890807 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f942c032890 con 0x7f94300ff440 2024-09-10T18:56:34.515 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.764+0000 7f94352b8640 1 -- 172.21.15.119:0/756890807 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f942c0317a0 con 0x7f94300ff440 2024-09-10T18:56:34.515 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.764+0000 7f943851e640 1 -- 172.21.15.119:0/756890807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94300ff440 msgr2=0x7f94300ff850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:34.515 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.764+0000 7f943851e640 1 --2- 172.21.15.119:0/756890807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94300ff440 0x7f94300ff850 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f942c009840 tx=0x7f942c0091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:34.515 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.764+0000 7f943851e640 1 -- 172.21.15.119:0/756890807 shutdown_connections 2024-09-10T18:56:34.515 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.765+0000 7f943851e640 1 --2- 172.21.15.119:0/756890807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94300ff440 0x7f94300ff850 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:34.515 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.765+0000 7f943851e640 1 -- 172.21.15.119:0/756890807 >> 172.21.15.119:0/756890807 conn(0x7f94300fabc0 msgr2=0x7f94300fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:34.515 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.765+0000 7f943851e640 1 -- 172.21.15.119:0/756890807 shutdown_connections 2024-09-10T18:56:34.515 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.765+0000 7f943851e640 1 -- 172.21.15.119:0/756890807 wait complete. 2024-09-10T18:56:34.515 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.765+0000 7f943851e640 1 Processor -- start 2024-09-10T18:56:34.515 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.765+0000 7f943851e640 1 -- start start 2024-09-10T18:56:34.515 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.765+0000 7f943851e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94300ff440 0x7f9430074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:34.516 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.765+0000 7f943851e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9430111f70 con 0x7f94300ff440 2024-09-10T18:56:34.516 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.765+0000 7f94362ba640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94300ff440 0x7f9430074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:34.516 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.765+0000 7f94362ba640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94300ff440 0x7f9430074b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53580/0 (socket says 172.21.15.119:53580) 2024-09-10T18:56:34.516 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.765+0000 7f94362ba640 1 -- 172.21.15.119:0/2582398219 learned_addr learned my addr 172.21.15.119:0/2582398219 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:34.516 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.766+0000 7f94362ba640 1 -- 172.21.15.119:0/2582398219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942c009450 con 0x7f94300ff440 2024-09-10T18:56:34.516 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.766+0000 7f94362ba640 1 --2- 172.21.15.119:0/2582398219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94300ff440 0x7f9430074b60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f942c0313e0 tx=0x7f942c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:34.516 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.766+0000 7f9426ffd640 1 -- 172.21.15.119:0/2582398219 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f942c031e70 con 0x7f94300ff440 2024-09-10T18:56:34.516 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.766+0000 7f9426ffd640 1 -- 172.21.15.119:0/2582398219 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f942c03c850 con 0x7f94300ff440 2024-09-10T18:56:34.516 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.766+0000 7f943851e640 1 -- 172.21.15.119:0/2582398219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94301120b0 con 0x7f94300ff440 2024-09-10T18:56:34.516 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.766+0000 7f9426ffd640 1 -- 172.21.15.119:0/2582398219 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f942c03a540 con 0x7f94300ff440 2024-09-10T18:56:34.516 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.766+0000 7f943851e640 1 -- 172.21.15.119:0/2582398219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9430112520 con 0x7f94300ff440 2024-09-10T18:56:34.516 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.766+0000 7f9426ffd640 1 -- 172.21.15.119:0/2582398219 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f942c03aa00 con 0x7f94300ff440 2024-09-10T18:56:34.517 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.766+0000 7f9426ffd640 1 -- 172.21.15.119:0/2582398219 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f942c046c00 con 0x7f94300ff440 2024-09-10T18:56:34.517 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.767+0000 7f943851e640 1 -- 172.21.15.119:0/2582398219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f943005ed70 con 0x7f94300ff440 2024-09-10T18:56:34.517 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.769+0000 7f9426ffd640 1 -- 172.21.15.119:0/2582398219 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f942c046e60 con 0x7f94300ff440 2024-09-10T18:56:34.517 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.826+0000 7f943851e640 1 -- 172.21.15.119:0/2582398219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9430101260 con 0x7f94300ff440 2024-09-10T18:56:34.517 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.826+0000 7f9426ffd640 1 -- 172.21.15.119:0/2582398219 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f942c037070 con 0x7f94300ff440 2024-09-10T18:56:34.517 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.827+0000 7f943851e640 1 -- 172.21.15.119:0/2582398219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94300ff440 msgr2=0x7f9430074b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:34.517 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.827+0000 7f943851e640 1 --2- 172.21.15.119:0/2582398219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94300ff440 0x7f9430074b60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f942c0313e0 tx=0x7f942c0091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:34.517 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.827+0000 7f943851e640 1 -- 172.21.15.119:0/2582398219 shutdown_connections 2024-09-10T18:56:34.517 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.827+0000 7f943851e640 1 --2- 172.21.15.119:0/2582398219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f94300ff440 0x7f9430074b60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:34.517 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.827+0000 7f943851e640 1 -- 172.21.15.119:0/2582398219 >> 172.21.15.119:0/2582398219 conn(0x7f94300fabc0 msgr2=0x7f94300fb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:34.517 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.827+0000 7f943851e640 1 -- 172.21.15.119:0/2582398219 shutdown_connections 2024-09-10T18:56:34.517 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:33.827+0000 7f943851e640 1 -- 172.21.15.119:0/2582398219 wait complete. 2024-09-10T18:56:34.517 INFO:teuthology.orchestra.run.smithi119.stdout:mgr not available, waiting (1/15)... 2024-09-10T18:56:35.316 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:34 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:34.985+0000 7f07097d5ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:56:35.317 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:35 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:35.048+0000 7f07097d5ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:56:36.037 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:35 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:35.751+0000 7f07097d5ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:56:36.037 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:35 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:35.859+0000 7f07097d5ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:56:36.037 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:35 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:35.977+0000 7f07097d5ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:56:36.316 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:36 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:36.035+0000 7f07097d5ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:56:36.316 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:36 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:36.159+0000 7f07097d5ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:56:36.317 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:36 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:36.225+0000 7f07097d5ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:56:36.317 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:36 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:36.284+0000 7f07097d5ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:56:36.690 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:36 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:36.416+0000 7f07097d5ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:56:36.690 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:36 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:36.478+0000 7f07097d5ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:56:36.690 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:36 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:36.549+0000 7f07097d5ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:56:36.690 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:36 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:36.629+0000 7f07097d5ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:56:36.690 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:36 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:36.688+0000 7f07097d5ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:56:36.967 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:36 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:36.751+0000 7f07097d5ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:56:37.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2133532427' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-10T18:56:37.662 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:37 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:37.516+0000 7f07097d5ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:56:37.666 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-10T18:56:37.667 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:56:37.667 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsid": "3afe9804-6fa6-11ef-bcea-c7b262605968", 2024-09-10T18:56:37.667 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "health": { 2024-09-10T18:56:37.667 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-10T18:56:37.667 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-10T18:56:37.667 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-10T18:56:37.667 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:37.667 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-10T18:56:37.668 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-10T18:56:37.668 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 0 2024-09-10T18:56:37.668 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:56:37.668 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-10T18:56:37.668 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "a" 2024-09-10T18:56:37.668 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:56:37.668 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-09-10T18:56:37.668 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-10T18:56:37.668 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:56:37.668 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-10T18:56:37.668 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-10T18:56:37.668 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:37.668 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-10T18:56:37.669 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:56:37.669 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-10T18:56:37.669 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-10T18:56:37.669 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-10T18:56:37.677 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-10T18:56:37.677 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-10T18:56:37.677 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-10T18:56:37.677 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:37.677 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-10T18:56:37.677 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-10T18:56:37.677 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-10T18:56:37.677 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-10T18:56:37.677 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-10T18:56:37.677 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-10T18:56:37.678 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-10T18:56:37.678 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-10T18:56:37.678 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-10T18:56:37.678 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:37.678 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-10T18:56:37.678 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:56:37.678 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-10T18:56:37.678 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-10T18:56:37.678 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:37.678 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-10T18:56:37.678 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-10T18:56:37.678 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-10T18:56:37.678 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-10T18:56:37.679 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-10T18:56:37.679 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-10T18:56:37.679 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "restful" 2024-09-10T18:56:37.679 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:56:37.679 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:56:37.679 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:37.679 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-10T18:56:37.679 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:56:37.679 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modified": "2024-09-10T18:56:26.299768+0000", 2024-09-10T18:56:37.679 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:56:37.679 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:37.679 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-10T18:56:37.679 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:56:37.680 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.965+0000 7f98d17c8640 1 Processor -- start 2024-09-10T18:56:37.680 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.965+0000 7f98d17c8640 1 -- start start 2024-09-10T18:56:37.680 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.965+0000 7f98d17c8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc074c40 0x7f98cc103190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:37.680 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.965+0000 7f98d17c8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc1036d0 con 0x7f98cc074c40 2024-09-10T18:56:37.680 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.965+0000 7f98caffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc074c40 0x7f98cc103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:37.680 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.966+0000 7f98caffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc074c40 0x7f98cc103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53592/0 (socket says 172.21.15.119:53592) 2024-09-10T18:56:37.680 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.966+0000 7f98caffd640 1 -- 172.21.15.119:0/49917828 learned_addr learned my addr 172.21.15.119:0/49917828 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:37.680 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.966+0000 7f98caffd640 1 -- 172.21.15.119:0/49917828 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc103810 con 0x7f98cc074c40 2024-09-10T18:56:37.680 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.967+0000 7f98caffd640 1 --2- 172.21.15.119:0/49917828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc074c40 0x7f98cc103190 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f98bc009840 tx=0x7f98bc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dceef733a5968425 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:37.681 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.967+0000 7f98ca7fc640 1 -- 172.21.15.119:0/49917828 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f98bc02add0 con 0x7f98cc074c40 2024-09-10T18:56:37.681 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.967+0000 7f98ca7fc640 1 -- 172.21.15.119:0/49917828 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f98bc032890 con 0x7f98cc074c40 2024-09-10T18:56:37.681 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.967+0000 7f98ca7fc640 1 -- 172.21.15.119:0/49917828 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f98bc0317a0 con 0x7f98cc074c40 2024-09-10T18:56:37.681 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.968+0000 7f98d17c8640 1 -- 172.21.15.119:0/49917828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc074c40 msgr2=0x7f98cc103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:37.681 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.968+0000 7f98d17c8640 1 --2- 172.21.15.119:0/49917828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc074c40 0x7f98cc103190 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f98bc009840 tx=0x7f98bc0091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:37.681 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.968+0000 7f98d17c8640 1 -- 172.21.15.119:0/49917828 shutdown_connections 2024-09-10T18:56:37.681 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.968+0000 7f98d17c8640 1 --2- 172.21.15.119:0/49917828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc074c40 0x7f98cc103190 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:37.681 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.968+0000 7f98d17c8640 1 -- 172.21.15.119:0/49917828 >> 172.21.15.119:0/49917828 conn(0x7f98cc0fabc0 msgr2=0x7f98cc0fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:37.681 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.968+0000 7f98d17c8640 1 -- 172.21.15.119:0/49917828 shutdown_connections 2024-09-10T18:56:37.681 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.968+0000 7f98d17c8640 1 -- 172.21.15.119:0/49917828 wait complete. 2024-09-10T18:56:37.681 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.968+0000 7f98d17c8640 1 Processor -- start 2024-09-10T18:56:37.681 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.969+0000 7f98d17c8640 1 -- start start 2024-09-10T18:56:37.681 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.969+0000 7f98d17c8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc074c40 0x7f98cc10f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:37.681 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.969+0000 7f98d17c8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc10f830 con 0x7f98cc074c40 2024-09-10T18:56:37.682 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.969+0000 7f98caffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc074c40 0x7f98cc10f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:37.682 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.969+0000 7f98caffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc074c40 0x7f98cc10f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53596/0 (socket says 172.21.15.119:53596) 2024-09-10T18:56:37.682 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.969+0000 7f98caffd640 1 -- 172.21.15.119:0/2133532427 learned_addr learned my addr 172.21.15.119:0/2133532427 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:37.682 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.969+0000 7f98caffd640 1 -- 172.21.15.119:0/2133532427 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc009450 con 0x7f98cc074c40 2024-09-10T18:56:37.682 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.969+0000 7f98caffd640 1 --2- 172.21.15.119:0/2133532427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc074c40 0x7f98cc10f2f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0313e0 tx=0x7f98bc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:37.682 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.970+0000 7f98c3fff640 1 -- 172.21.15.119:0/2133532427 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f98bc032cd0 con 0x7f98cc074c40 2024-09-10T18:56:37.682 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.970+0000 7f98d17c8640 1 -- 172.21.15.119:0/2133532427 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc10fa30 con 0x7f98cc074c40 2024-09-10T18:56:37.682 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.970+0000 7f98c3fff640 1 -- 172.21.15.119:0/2133532427 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f98bc03c850 con 0x7f98cc074c40 2024-09-10T18:56:37.682 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.970+0000 7f98c3fff640 1 -- 172.21.15.119:0/2133532427 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f98bc03a3b0 con 0x7f98cc074c40 2024-09-10T18:56:37.682 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.970+0000 7f98d17c8640 1 -- 172.21.15.119:0/2133532427 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc10fea0 con 0x7f98cc074c40 2024-09-10T18:56:37.682 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.970+0000 7f98c3fff640 1 -- 172.21.15.119:0/2133532427 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f98bc03a900 con 0x7f98cc074c40 2024-09-10T18:56:37.682 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.970+0000 7f98c3fff640 1 -- 172.21.15.119:0/2133532427 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f98bc046b70 con 0x7f98cc074c40 2024-09-10T18:56:37.683 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.971+0000 7f98d17c8640 1 -- 172.21.15.119:0/2133532427 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98cc04b720 con 0x7f98cc074c40 2024-09-10T18:56:37.683 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:36.973+0000 7f98c3fff640 1 -- 172.21.15.119:0/2133532427 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71553 (secure 0 0 0) 0x7f98cc10fea0 con 0x7f98cc074c40 2024-09-10T18:56:37.683 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:37.031+0000 7f98d17c8640 1 -- 172.21.15.119:0/2133532427 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f98cc110660 con 0x7f98cc074c40 2024-09-10T18:56:37.683 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:37.032+0000 7f98c3fff640 1 -- 172.21.15.119:0/2133532427 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f98bc037070 con 0x7f98cc074c40 2024-09-10T18:56:37.683 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:37.033+0000 7f98d17c8640 1 -- 172.21.15.119:0/2133532427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc074c40 msgr2=0x7f98cc10f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:37.683 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:37.033+0000 7f98d17c8640 1 --2- 172.21.15.119:0/2133532427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc074c40 0x7f98cc10f2f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0313e0 tx=0x7f98bc0091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:37.683 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:37.033+0000 7f98d17c8640 1 -- 172.21.15.119:0/2133532427 shutdown_connections 2024-09-10T18:56:37.683 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:37.033+0000 7f98d17c8640 1 --2- 172.21.15.119:0/2133532427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc074c40 0x7f98cc10f2f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:37.683 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:37.033+0000 7f98d17c8640 1 -- 172.21.15.119:0/2133532427 >> 172.21.15.119:0/2133532427 conn(0x7f98cc0fabc0 msgr2=0x7f98cc0fb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:37.683 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:37.033+0000 7f98d17c8640 1 -- 172.21.15.119:0/2133532427 shutdown_connections 2024-09-10T18:56:37.683 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:37.033+0000 7f98d17c8640 1 -- 172.21.15.119:0/2133532427 wait complete. 2024-09-10T18:56:37.683 INFO:teuthology.orchestra.run.smithi119.stdout:mgr not available, waiting (2/15)... 2024-09-10T18:56:38.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:38 smithi119 ceph-mon[29577]: Activating manager daemon y 2024-09-10T18:56:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:38 smithi119 ceph-mon[29577]: mgrmap e2: y(active, starting, since 0.00283043s) 2024-09-10T18:56:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:38 smithi119 ceph-mon[29577]: from='mgr.14100 172.21.15.119:0/330513660' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:56:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:38 smithi119 ceph-mon[29577]: from='mgr.14100 172.21.15.119:0/330513660' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:56:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:38 smithi119 ceph-mon[29577]: from='mgr.14100 172.21.15.119:0/330513660' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:56:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:38 smithi119 ceph-mon[29577]: from='mgr.14100 172.21.15.119:0/330513660' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:56:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:38 smithi119 ceph-mon[29577]: from='mgr.14100 172.21.15.119:0/330513660' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:56:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:38 smithi119 ceph-mon[29577]: Manager daemon y is now available 2024-09-10T18:56:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:38 smithi119 ceph-mon[29577]: from='mgr.14100 172.21.15.119:0/330513660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:56:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:38 smithi119 ceph-mon[29577]: from='mgr.14100 172.21.15.119:0/330513660' entity='mgr.y' 2024-09-10T18:56:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:38 smithi119 ceph-mon[29577]: from='mgr.14100 172.21.15.119:0/330513660' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:56:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:38 smithi119 ceph-mon[29577]: from='mgr.14100 172.21.15.119:0/330513660' entity='mgr.y' 2024-09-10T18:56:38.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:38 smithi119 ceph-mon[29577]: from='mgr.14100 172.21.15.119:0/330513660' entity='mgr.y' 2024-09-10T18:56:39.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:39 smithi119 ceph-mon[29577]: mgrmap e3: y(active, since 1.00677s) 2024-09-10T18:56:40.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:40 smithi119 ceph-mon[29577]: mgrmap e4: y(active, since 2s) 2024-09-10T18:56:40.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1914668018' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-10T18:56:40.931 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-10T18:56:40.931 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:56:40.931 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsid": "3afe9804-6fa6-11ef-bcea-c7b262605968", 2024-09-10T18:56:40.931 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "health": { 2024-09-10T18:56:40.931 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-10T18:56:40.932 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-10T18:56:40.932 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-10T18:56:40.932 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:40.932 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-10T18:56:40.932 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-10T18:56:40.932 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 0 2024-09-10T18:56:40.932 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:56:40.932 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-10T18:56:40.932 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "a" 2024-09-10T18:56:40.932 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:56:40.932 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-10T18:56:40.932 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-10T18:56:40.932 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:56:40.933 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-10T18:56:40.933 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-10T18:56:40.933 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:40.933 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-10T18:56:40.933 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:56:40.933 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-10T18:56:40.933 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-10T18:56:40.933 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-10T18:56:40.942 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-10T18:56:40.942 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-10T18:56:40.943 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-10T18:56:40.943 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:40.943 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-10T18:56:40.943 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-10T18:56:40.943 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-10T18:56:40.943 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-10T18:56:40.943 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-10T18:56:40.943 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-10T18:56:40.943 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-10T18:56:40.943 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-10T18:56:40.943 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-10T18:56:40.943 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:40.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-10T18:56:40.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:56:40.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-10T18:56:40.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-10T18:56:40.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:40.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-10T18:56:40.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T18:56:40.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-10T18:56:40.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-10T18:56:40.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-10T18:56:40.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-10T18:56:40.944 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "restful" 2024-09-10T18:56:40.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ], 2024-09-10T18:56:40.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:56:40.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:40.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-10T18:56:40.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T18:56:40.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "modified": "2024-09-10T18:56:26.299768+0000", 2024-09-10T18:56:40.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T18:56:40.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout }, 2024-09-10T18:56:40.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-10T18:56:40.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:56:40.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.099+0000 7f757d56e640 1 Processor -- start 2024-09-10T18:56:40.945 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.099+0000 7f757d56e640 1 -- start start 2024-09-10T18:56:40.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.099+0000 7f757d56e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75780754e0 0x7f7578105630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:40.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.099+0000 7f757d56e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7578101020 con 0x7f75780754e0 2024-09-10T18:56:40.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.099+0000 7f7576ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75780754e0 0x7f7578105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:40.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.099+0000 7f7576ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75780754e0 0x7f7578105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53652/0 (socket says 172.21.15.119:53652) 2024-09-10T18:56:40.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.099+0000 7f7576ffd640 1 -- 172.21.15.119:0/3900889444 learned_addr learned my addr 172.21.15.119:0/3900889444 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:40.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.100+0000 7f7576ffd640 1 -- 172.21.15.119:0/3900889444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7578105b70 con 0x7f75780754e0 2024-09-10T18:56:40.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.100+0000 7f7576ffd640 1 --2- 172.21.15.119:0/3900889444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75780754e0 0x7f7578105630 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7568009840 tx=0x7f75680091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1258397ff2ed7378 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:40.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.100+0000 7f75767fc640 1 -- 172.21.15.119:0/3900889444 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f756802add0 con 0x7f75780754e0 2024-09-10T18:56:40.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.100+0000 7f75767fc640 1 -- 172.21.15.119:0/3900889444 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f7568032890 con 0x7f75780754e0 2024-09-10T18:56:40.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.100+0000 7f75767fc640 1 -- 172.21.15.119:0/3900889444 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f75680317a0 con 0x7f75780754e0 2024-09-10T18:56:40.946 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.101+0000 7f757d56e640 1 -- 172.21.15.119:0/3900889444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75780754e0 msgr2=0x7f7578105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:40.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.101+0000 7f757d56e640 1 --2- 172.21.15.119:0/3900889444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75780754e0 0x7f7578105630 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7568009840 tx=0x7f75680091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:40.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.101+0000 7f757d56e640 1 -- 172.21.15.119:0/3900889444 shutdown_connections 2024-09-10T18:56:40.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.101+0000 7f757d56e640 1 --2- 172.21.15.119:0/3900889444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75780754e0 0x7f7578105630 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:40.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.101+0000 7f757d56e640 1 -- 172.21.15.119:0/3900889444 >> 172.21.15.119:0/3900889444 conn(0x7f75780fad60 msgr2=0x7f75780fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:40.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.101+0000 7f757d56e640 1 -- 172.21.15.119:0/3900889444 shutdown_connections 2024-09-10T18:56:40.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.101+0000 7f757d56e640 1 -- 172.21.15.119:0/3900889444 wait complete. 2024-09-10T18:56:40.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.102+0000 7f757d56e640 1 Processor -- start 2024-09-10T18:56:40.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.102+0000 7f757d56e640 1 -- start start 2024-09-10T18:56:40.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.102+0000 7f757d56e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75780754e0 0x7f75781114a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:40.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.102+0000 7f757d56e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75781119e0 con 0x7f75780754e0 2024-09-10T18:56:40.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.102+0000 7f7576ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75780754e0 0x7f75781114a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:40.947 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.102+0000 7f7576ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75780754e0 0x7f75781114a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53666/0 (socket says 172.21.15.119:53666) 2024-09-10T18:56:40.948 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.102+0000 7f7576ffd640 1 -- 172.21.15.119:0/1914668018 learned_addr learned my addr 172.21.15.119:0/1914668018 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:40.948 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.102+0000 7f7576ffd640 1 -- 172.21.15.119:0/1914668018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7568009450 con 0x7f75780754e0 2024-09-10T18:56:40.948 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.102+0000 7f7576ffd640 1 --2- 172.21.15.119:0/1914668018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75780754e0 0x7f75781114a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f75680313e0 tx=0x7f75680091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:40.948 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.102+0000 7f756f7fe640 1 -- 172.21.15.119:0/1914668018 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7568032cd0 con 0x7f75780754e0 2024-09-10T18:56:40.948 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.103+0000 7f756f7fe640 1 -- 172.21.15.119:0/1914668018 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f756803c850 con 0x7f75780754e0 2024-09-10T18:56:40.948 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.103+0000 7f757d56e640 1 -- 172.21.15.119:0/1914668018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7578111be0 con 0x7f75780754e0 2024-09-10T18:56:40.948 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.103+0000 7f756f7fe640 1 -- 172.21.15.119:0/1914668018 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f756803a3b0 con 0x7f75780754e0 2024-09-10T18:56:40.948 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.103+0000 7f757d56e640 1 -- 172.21.15.119:0/1914668018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7578112770 con 0x7f75780754e0 2024-09-10T18:56:40.948 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.103+0000 7f756f7fe640 1 -- 172.21.15.119:0/1914668018 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 4) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f756803a620 con 0x7f75780754e0 2024-09-10T18:56:40.948 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.103+0000 7f757d56e640 1 -- 172.21.15.119:0/1914668018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75781120b0 con 0x7f75780754e0 2024-09-10T18:56:40.948 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.103+0000 7f756f7fe640 1 --2- 172.21.15.119:0/1914668018 >> v2:172.21.15.119:6800/4184327284 conn(0x7f7564039430 0x7f756403b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:40.948 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.104+0000 7f756f7fe640 1 -- 172.21.15.119:0/1914668018 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f756806cd20 con 0x7f75780754e0 2024-09-10T18:56:40.949 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.104+0000 7f756ffff640 1 --2- 172.21.15.119:0/1914668018 >> v2:172.21.15.119:6800/4184327284 conn(0x7f7564039430 0x7f756403b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:40.949 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.104+0000 7f756ffff640 1 --2- 172.21.15.119:0/1914668018 >> v2:172.21.15.119:6800/4184327284 conn(0x7f7564039430 0x7f756403b8e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7560006fd0 tx=0x7f7560006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:40.949 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.108+0000 7f756f7fe640 1 -- 172.21.15.119:0/1914668018 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f75781120b0 con 0x7f75780754e0 2024-09-10T18:56:40.949 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.320+0000 7f757d56e640 1 -- 172.21.15.119:0/1914668018 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f75781120b0 con 0x7f75780754e0 2024-09-10T18:56:40.949 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.321+0000 7f756f7fe640 1 -- 172.21.15.119:0/1914668018 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f75781120b0 con 0x7f75780754e0 2024-09-10T18:56:40.949 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.322+0000 7f757d56e640 1 -- 172.21.15.119:0/1914668018 >> v2:172.21.15.119:6800/4184327284 conn(0x7f7564039430 msgr2=0x7f756403b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:40.949 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.322+0000 7f757d56e640 1 --2- 172.21.15.119:0/1914668018 >> v2:172.21.15.119:6800/4184327284 conn(0x7f7564039430 0x7f756403b8e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7560006fd0 tx=0x7f7560006db0 comp rx=0 tx=0).stop 2024-09-10T18:56:40.949 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.323+0000 7f757d56e640 1 -- 172.21.15.119:0/1914668018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75780754e0 msgr2=0x7f75781114a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:40.949 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.323+0000 7f757d56e640 1 --2- 172.21.15.119:0/1914668018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75780754e0 0x7f75781114a0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f75680313e0 tx=0x7f75680091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:40.949 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.323+0000 7f757d56e640 1 -- 172.21.15.119:0/1914668018 shutdown_connections 2024-09-10T18:56:40.949 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.323+0000 7f757d56e640 1 --2- 172.21.15.119:0/1914668018 >> v2:172.21.15.119:6800/4184327284 conn(0x7f7564039430 0x7f756403b8e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:40.950 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.323+0000 7f757d56e640 1 --2- 172.21.15.119:0/1914668018 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75780754e0 0x7f75781114a0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:40.950 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.323+0000 7f757d56e640 1 -- 172.21.15.119:0/1914668018 >> 172.21.15.119:0/1914668018 conn(0x7f75780fad60 msgr2=0x7f75780fb8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:40.950 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.323+0000 7f757d56e640 1 -- 172.21.15.119:0/1914668018 shutdown_connections 2024-09-10T18:56:40.950 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:40.323+0000 7f757d56e640 1 -- 172.21.15.119:0/1914668018 wait complete. 2024-09-10T18:56:40.950 INFO:teuthology.orchestra.run.smithi119.stdout:mgr is available 2024-09-10T18:56:41.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2147449291' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-10T18:56:47.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-10T18:56:47.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [global] 2024-09-10T18:56:47.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout fsid = 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:56:47.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-10T18:56:47.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-10T18:56:47.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-10T18:56:47.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-10T18:56:47.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-10T18:56:47.460 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-10T18:56:47.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-10T18:56:47.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-10T18:56:47.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 2024-09-10T18:56:47.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout [osd] 2024-09-10T18:56:47.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-10T18:56:47.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-10T18:56:47.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.378+0000 7fc740e8d640 1 Processor -- start 2024-09-10T18:56:47.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.378+0000 7fc740e8d640 1 -- start start 2024-09-10T18:56:47.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.378+0000 7fc740e8d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc73c1005b0 0x7fc73c1009c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:47.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.378+0000 7fc740e8d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c100f00 con 0x7fc73c1005b0 2024-09-10T18:56:47.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.379+0000 7fc73a59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc73c1005b0 0x7fc73c1009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:47.461 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.379+0000 7fc73a59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc73c1005b0 0x7fc73c1009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53672/0 (socket says 172.21.15.119:53672) 2024-09-10T18:56:47.462 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.379+0000 7fc73a59c640 1 -- 172.21.15.119:0/3035510488 learned_addr learned my addr 172.21.15.119:0/3035510488 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:47.462 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.379+0000 7fc73a59c640 1 -- 172.21.15.119:0/3035510488 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c101040 con 0x7fc73c1005b0 2024-09-10T18:56:47.462 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.379+0000 7fc73a59c640 1 --2- 172.21.15.119:0/3035510488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc73c1005b0 0x7fc73c1009c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc72c009840 tx=0x7fc72c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff69c706f58330d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:47.462 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.380+0000 7fc73959a640 1 -- 172.21.15.119:0/3035510488 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc72c02add0 con 0x7fc73c1005b0 2024-09-10T18:56:47.462 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.380+0000 7fc73959a640 1 -- 172.21.15.119:0/3035510488 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7fc72c032890 con 0x7fc73c1005b0 2024-09-10T18:56:47.462 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.380+0000 7fc73959a640 1 -- 172.21.15.119:0/3035510488 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc72c0317a0 con 0x7fc73c1005b0 2024-09-10T18:56:47.462 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.380+0000 7fc740e8d640 1 -- 172.21.15.119:0/3035510488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc73c1005b0 msgr2=0x7fc73c1009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:47.462 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.380+0000 7fc740e8d640 1 --2- 172.21.15.119:0/3035510488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc73c1005b0 0x7fc73c1009c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc72c009840 tx=0x7fc72c0091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:47.462 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.380+0000 7fc740e8d640 1 -- 172.21.15.119:0/3035510488 shutdown_connections 2024-09-10T18:56:47.462 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.380+0000 7fc740e8d640 1 --2- 172.21.15.119:0/3035510488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc73c1005b0 0x7fc73c1009c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:47.462 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.380+0000 7fc740e8d640 1 -- 172.21.15.119:0/3035510488 >> 172.21.15.119:0/3035510488 conn(0x7fc73c0fbe10 msgr2=0x7fc73c0fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:47.462 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.380+0000 7fc740e8d640 1 -- 172.21.15.119:0/3035510488 shutdown_connections 2024-09-10T18:56:47.462 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.380+0000 7fc740e8d640 1 -- 172.21.15.119:0/3035510488 wait complete. 2024-09-10T18:56:47.463 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.381+0000 7fc740e8d640 1 Processor -- start 2024-09-10T18:56:47.463 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.381+0000 7fc740e8d640 1 -- start start 2024-09-10T18:56:47.463 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.381+0000 7fc740e8d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc73c1005b0 0x7fc73c10f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:47.463 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.381+0000 7fc740e8d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c10f780 con 0x7fc73c1005b0 2024-09-10T18:56:47.463 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.381+0000 7fc73a59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc73c1005b0 0x7fc73c10f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:47.463 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.381+0000 7fc73a59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc73c1005b0 0x7fc73c10f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53688/0 (socket says 172.21.15.119:53688) 2024-09-10T18:56:47.463 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.381+0000 7fc73a59c640 1 -- 172.21.15.119:0/2147449291 learned_addr learned my addr 172.21.15.119:0/2147449291 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:47.463 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.381+0000 7fc73a59c640 1 -- 172.21.15.119:0/2147449291 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72c009450 con 0x7fc73c1005b0 2024-09-10T18:56:47.463 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.382+0000 7fc73a59c640 1 --2- 172.21.15.119:0/2147449291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc73c1005b0 0x7fc73c10f240 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0313e0 tx=0x7fc72c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:47.463 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.382+0000 7fc72affd640 1 -- 172.21.15.119:0/2147449291 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc72c031e70 con 0x7fc73c1005b0 2024-09-10T18:56:47.463 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.382+0000 7fc72affd640 1 -- 172.21.15.119:0/2147449291 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7fc72c03c850 con 0x7fc73c1005b0 2024-09-10T18:56:47.463 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.382+0000 7fc740e8d640 1 -- 172.21.15.119:0/2147449291 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73c10f980 con 0x7fc73c1005b0 2024-09-10T18:56:47.463 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.382+0000 7fc72affd640 1 -- 172.21.15.119:0/2147449291 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc72c03a540 con 0x7fc73c1005b0 2024-09-10T18:56:47.463 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.382+0000 7fc740e8d640 1 -- 172.21.15.119:0/2147449291 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73c10fdf0 con 0x7fc73c1005b0 2024-09-10T18:56:47.464 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.383+0000 7fc72affd640 1 -- 172.21.15.119:0/2147449291 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 4) v1 ==== 45446+0+0 (secure 0 0 0) 0x7fc72c03a7c0 con 0x7fc73c1005b0 2024-09-10T18:56:47.464 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.383+0000 7fc72affd640 1 --2- 172.21.15.119:0/2147449291 >> v2:172.21.15.119:6800/4184327284 conn(0x7fc7240393e0 0x7fc72403b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:47.464 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.383+0000 7fc740e8d640 1 -- 172.21.15.119:0/2147449291 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73c10fdf0 con 0x7fc73c1005b0 2024-09-10T18:56:47.464 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.383+0000 7fc72affd640 1 -- 172.21.15.119:0/2147449291 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc72c06caa0 con 0x7fc73c1005b0 2024-09-10T18:56:47.464 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.383+0000 7fc739d9b640 1 --2- 172.21.15.119:0/2147449291 >> v2:172.21.15.119:6800/4184327284 conn(0x7fc7240393e0 0x7fc72403b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:47.464 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.383+0000 7fc739d9b640 1 --2- 172.21.15.119:0/2147449291 >> v2:172.21.15.119:6800/4184327284 conn(0x7fc7240393e0 0x7fc72403b890 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc730006fd0 tx=0x7fc730006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:47.464 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.387+0000 7fc72affd640 1 -- 172.21.15.119:0/2147449291 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fc73c10fdf0 con 0x7fc73c1005b0 2024-09-10T18:56:47.464 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.545+0000 7fc740e8d640 1 -- 172.21.15.119:0/2147449291 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fc73c04b720 con 0x7fc73c1005b0 2024-09-10T18:56:47.464 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.546+0000 7fc72affd640 1 -- 172.21.15.119:0/2147449291 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+318 (secure 0 0 0) 0x7fc72c055020 con 0x7fc73c1005b0 2024-09-10T18:56:47.464 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.548+0000 7fc740e8d640 1 -- 172.21.15.119:0/2147449291 >> v2:172.21.15.119:6800/4184327284 conn(0x7fc7240393e0 msgr2=0x7fc72403b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:47.464 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.548+0000 7fc740e8d640 1 --2- 172.21.15.119:0/2147449291 >> v2:172.21.15.119:6800/4184327284 conn(0x7fc7240393e0 0x7fc72403b890 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc730006fd0 tx=0x7fc730006db0 comp rx=0 tx=0).stop 2024-09-10T18:56:47.464 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.548+0000 7fc740e8d640 1 -- 172.21.15.119:0/2147449291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc73c1005b0 msgr2=0x7fc73c10f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:47.464 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.548+0000 7fc740e8d640 1 --2- 172.21.15.119:0/2147449291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc73c1005b0 0x7fc73c10f240 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc72c0313e0 tx=0x7fc72c0091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:47.465 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.548+0000 7fc740e8d640 1 -- 172.21.15.119:0/2147449291 shutdown_connections 2024-09-10T18:56:47.465 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.548+0000 7fc740e8d640 1 --2- 172.21.15.119:0/2147449291 >> v2:172.21.15.119:6800/4184327284 conn(0x7fc7240393e0 0x7fc72403b890 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:47.465 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.548+0000 7fc740e8d640 1 --2- 172.21.15.119:0/2147449291 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc73c1005b0 0x7fc73c10f240 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:47.465 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.548+0000 7fc740e8d640 1 -- 172.21.15.119:0/2147449291 >> 172.21.15.119:0/2147449291 conn(0x7fc73c0fbe10 msgr2=0x7fc73c0fc870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:47.465 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.549+0000 7fc740e8d640 1 -- 172.21.15.119:0/2147449291 shutdown_connections 2024-09-10T18:56:47.465 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:41.549+0000 7fc740e8d640 1 -- 172.21.15.119:0/2147449291 wait complete. 2024-09-10T18:56:47.465 INFO:teuthology.orchestra.run.smithi119.stdout:Enabling cephadm module... 2024-09-10T18:56:50.994 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3488535699' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-10T18:56:50.994 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:50 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ignoring --setuser ceph since I am not root 2024-09-10T18:56:50.994 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:50 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ignoring --setgroup ceph since I am not root 2024-09-10T18:56:50.994 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:50 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:50.873+0000 7f15cfa5aec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:56:50.994 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:50 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:50.927+0000 7f15cfa5aec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:56:50.994 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:50 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:50.991+0000 7f15cfa5aec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:56:51.253 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:51 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:51.116+0000 7f15cfa5aec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:56:51.253 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:51 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:51.251+0000 7f15cfa5aec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:56:51.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3488535699' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-10T18:56:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:51 smithi119 ceph-mon[29577]: mgrmap e5: y(active, since 13s) 2024-09-10T18:56:51.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:51 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:51.507+0000 7f15cfa5aec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:56:51.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:51 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:51.567+0000 7f15cfa5aec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:56:51.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:51 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:51.727+0000 7f15cfa5aec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:56:52.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:52.564+0000 7f15cfa5aec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:56:52.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:52.627+0000 7f15cfa5aec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:56:53.566 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:53 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:53.338+0000 7f15cfa5aec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:56:53.566 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:53 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:53.449+0000 7f15cfa5aec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:56:53.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.482+0000 7f95fbd1f640 1 Processor -- start 2024-09-10T18:56:53.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.483+0000 7f95fbd1f640 1 -- start start 2024-09-10T18:56:53.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.483+0000 7f95fbd1f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4077470 0x7f95f4075ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:53.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.483+0000 7f95fbd1f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f4077880 con 0x7f95f4077470 2024-09-10T18:56:53.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.483+0000 7f95f9abb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4077470 0x7f95f4075ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:53.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.483+0000 7f95f9abb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4077470 0x7f95f4075ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40566/0 (socket says 172.21.15.119:40566) 2024-09-10T18:56:53.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.483+0000 7f95f9abb640 1 -- 172.21.15.119:0/3523076325 learned_addr learned my addr 172.21.15.119:0/3523076325 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:53.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.484+0000 7f95f9abb640 1 -- 172.21.15.119:0/3523076325 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f40779c0 con 0x7f95f4077470 2024-09-10T18:56:53.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.484+0000 7f95f9abb640 1 --2- 172.21.15.119:0/3523076325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4077470 0x7f95f4075ae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f95e4009840 tx=0x7f95e40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f4e51b72090161 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:53.576 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.485+0000 7f95f8ab9640 1 -- 172.21.15.119:0/3523076325 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95e402add0 con 0x7f95f4077470 2024-09-10T18:56:53.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.485+0000 7f95f8ab9640 1 -- 172.21.15.119:0/3523076325 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f95e4032890 con 0x7f95f4077470 2024-09-10T18:56:53.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.485+0000 7f95f8ab9640 1 -- 172.21.15.119:0/3523076325 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95e40317a0 con 0x7f95f4077470 2024-09-10T18:56:53.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.485+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3523076325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4077470 msgr2=0x7f95f4075ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:53.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.485+0000 7f95fbd1f640 1 --2- 172.21.15.119:0/3523076325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4077470 0x7f95f4075ae0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f95e4009840 tx=0x7f95e40091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:53.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.485+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3523076325 shutdown_connections 2024-09-10T18:56:53.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.485+0000 7f95fbd1f640 1 --2- 172.21.15.119:0/3523076325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4077470 0x7f95f4075ae0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:53.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.485+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3523076325 >> 172.21.15.119:0/3523076325 conn(0x7f95f40fbe10 msgr2=0x7f95f40fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:53.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.485+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3523076325 shutdown_connections 2024-09-10T18:56:53.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.485+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3523076325 wait complete. 2024-09-10T18:56:53.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.486+0000 7f95fbd1f640 1 Processor -- start 2024-09-10T18:56:53.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.486+0000 7f95fbd1f640 1 -- start start 2024-09-10T18:56:53.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.486+0000 7f95fbd1f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4077470 0x7f95f41114c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:53.577 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.486+0000 7f95fbd1f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95f4111a00 con 0x7f95f4077470 2024-09-10T18:56:53.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.486+0000 7f95f9abb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4077470 0x7f95f41114c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:53.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.486+0000 7f95f9abb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4077470 0x7f95f41114c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40570/0 (socket says 172.21.15.119:40570) 2024-09-10T18:56:53.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.486+0000 7f95f9abb640 1 -- 172.21.15.119:0/3488535699 learned_addr learned my addr 172.21.15.119:0/3488535699 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:53.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.486+0000 7f95f9abb640 1 -- 172.21.15.119:0/3488535699 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e4009450 con 0x7f95f4077470 2024-09-10T18:56:53.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.486+0000 7f95f9abb640 1 --2- 172.21.15.119:0/3488535699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4077470 0x7f95f41114c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f95e40313e0 tx=0x7f95e40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:53.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.487+0000 7f95ea7fc640 1 -- 172.21.15.119:0/3488535699 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95e4032cd0 con 0x7f95f4077470 2024-09-10T18:56:53.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.487+0000 7f95ea7fc640 1 -- 172.21.15.119:0/3488535699 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f95e403c850 con 0x7f95f4077470 2024-09-10T18:56:53.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.487+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3488535699 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f4111c00 con 0x7f95f4077470 2024-09-10T18:56:53.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.487+0000 7f95ea7fc640 1 -- 172.21.15.119:0/3488535699 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95e403a3b0 con 0x7f95f4077470 2024-09-10T18:56:53.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.487+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3488535699 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f4112070 con 0x7f95f4077470 2024-09-10T18:56:53.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.488+0000 7f95ea7fc640 1 -- 172.21.15.119:0/3488535699 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 4) v1 ==== 45446+0+0 (secure 0 0 0) 0x7f95e403a620 con 0x7f95f4077470 2024-09-10T18:56:53.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.488+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3488535699 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f404b720 con 0x7f95f4077470 2024-09-10T18:56:53.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.488+0000 7f95ea7fc640 1 --2- 172.21.15.119:0/3488535699 >> v2:172.21.15.119:6800/4184327284 conn(0x7f95e0039430 0x7f95e003b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:53.578 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.488+0000 7f95ea7fc640 1 -- 172.21.15.119:0/3488535699 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f95e406c9c0 con 0x7f95f4077470 2024-09-10T18:56:53.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.488+0000 7f95f92ba640 1 --2- 172.21.15.119:0/3488535699 >> v2:172.21.15.119:6800/4184327284 conn(0x7f95e0039430 0x7f95e003b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:53.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.488+0000 7f95f92ba640 1 --2- 172.21.15.119:0/3488535699 >> v2:172.21.15.119:6800/4184327284 conn(0x7f95e0039430 0x7f95e003b8e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f95f0006fd0 tx=0x7f95f0006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:53.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.492+0000 7f95ea7fc640 1 -- 172.21.15.119:0/3488535699 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f95e4046460 con 0x7f95f4077470 2024-09-10T18:56:53.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.691+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3488535699 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f95f4112070 con 0x7f95f4077470 2024-09-10T18:56:53.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.744+0000 7f95ea7fc640 1 -- 172.21.15.119:0/3488535699 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 5) v1 ==== 45457+0+0 (secure 0 0 0) 0x7f95e403a8d0 con 0x7f95f4077470 2024-09-10T18:56:53.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.745+0000 7f95ea7fc640 1 -- 172.21.15.119:0/3488535699 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f95e406c3d0 con 0x7f95f4077470 2024-09-10T18:56:53.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.747+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3488535699 >> v2:172.21.15.119:6800/4184327284 conn(0x7f95e0039430 msgr2=0x7f95e003b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:53.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.747+0000 7f95fbd1f640 1 --2- 172.21.15.119:0/3488535699 >> v2:172.21.15.119:6800/4184327284 conn(0x7f95e0039430 0x7f95e003b8e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f95f0006fd0 tx=0x7f95f0006db0 comp rx=0 tx=0).stop 2024-09-10T18:56:53.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.747+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3488535699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4077470 msgr2=0x7f95f41114c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:53.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.747+0000 7f95fbd1f640 1 --2- 172.21.15.119:0/3488535699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4077470 0x7f95f41114c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f95e40313e0 tx=0x7f95e40091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:53.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.747+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3488535699 shutdown_connections 2024-09-10T18:56:53.579 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.747+0000 7f95fbd1f640 1 --2- 172.21.15.119:0/3488535699 >> v2:172.21.15.119:6800/4184327284 conn(0x7f95e0039430 0x7f95e003b8e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:53.580 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.747+0000 7f95fbd1f640 1 --2- 172.21.15.119:0/3488535699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95f4077470 0x7f95f41114c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:53.580 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.747+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3488535699 >> 172.21.15.119:0/3488535699 conn(0x7f95f40fbe10 msgr2=0x7f95f40fca20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:53.580 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.748+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3488535699 shutdown_connections 2024-09-10T18:56:53.580 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:50.748+0000 7f95fbd1f640 1 -- 172.21.15.119:0/3488535699 wait complete. 2024-09-10T18:56:53.821 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:53 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:53.573+0000 7f15cfa5aec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:56:53.821 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:53 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:53.637+0000 7f15cfa5aec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:56:53.821 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:53 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:53.763+0000 7f15cfa5aec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:56:54.071 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:53 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:53.832+0000 7f15cfa5aec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:56:54.071 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:53 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:53.893+0000 7f15cfa5aec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:56:54.071 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:54 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:54.029+0000 7f15cfa5aec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:56:54.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4075951049' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-10T18:56:54.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:54 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:54.097+0000 7f15cfa5aec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:56:54.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:54 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:54.162+0000 7f15cfa5aec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:56:54.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:54 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:54.245+0000 7f15cfa5aec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:56:54.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:54 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:54.310+0000 7f15cfa5aec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:56:54.369 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:54 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:54.367+0000 7f15cfa5aec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:56:54.824 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:56:54.824 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-10T18:56:54.824 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T18:56:54.824 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-10T18:56:54.824 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-10T18:56:54.824 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:56:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.054+0000 7fd4acd80640 1 Processor -- start 2024-09-10T18:56:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.054+0000 7fd4acd80640 1 -- start start 2024-09-10T18:56:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.054+0000 7fd4acd80640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4a8077480 0x7fd4a8075aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.054+0000 7fd4acd80640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a8077890 con 0x7fd4a8077480 2024-09-10T18:56:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.054+0000 7fd4a659c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4a8077480 0x7fd4a8075aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.054+0000 7fd4a659c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4a8077480 0x7fd4a8075aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49136/0 (socket says 172.21.15.119:49136) 2024-09-10T18:56:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.054+0000 7fd4a659c640 1 -- 172.21.15.119:0/1565175459 learned_addr learned my addr 172.21.15.119:0/1565175459 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.054+0000 7fd4a659c640 1 -- 172.21.15.119:0/1565175459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4a80779d0 con 0x7fd4a8077480 2024-09-10T18:56:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.055+0000 7fd4a659c640 1 --2- 172.21.15.119:0/1565175459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4a8077480 0x7fd4a8075aa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd498009840 tx=0x7fd4980091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd4330ca178d425 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.055+0000 7fd4a559a640 1 -- 172.21.15.119:0/1565175459 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd49802add0 con 0x7fd4a8077480 2024-09-10T18:56:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.055+0000 7fd4a559a640 1 -- 172.21.15.119:0/1565175459 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7fd498032890 con 0x7fd4a8077480 2024-09-10T18:56:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.055+0000 7fd4a559a640 1 -- 172.21.15.119:0/1565175459 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4980317a0 con 0x7fd4a8077480 2024-09-10T18:56:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.056+0000 7fd4acd80640 1 -- 172.21.15.119:0/1565175459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4a8077480 msgr2=0x7fd4a8075aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.056+0000 7fd4acd80640 1 --2- 172.21.15.119:0/1565175459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4a8077480 0x7fd4a8075aa0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd498009840 tx=0x7fd4980091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.056+0000 7fd4acd80640 1 -- 172.21.15.119:0/1565175459 shutdown_connections 2024-09-10T18:56:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.056+0000 7fd4acd80640 1 --2- 172.21.15.119:0/1565175459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4a8077480 0x7fd4a8075aa0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.056+0000 7fd4acd80640 1 -- 172.21.15.119:0/1565175459 >> 172.21.15.119:0/1565175459 conn(0x7fd4a80fbea0 msgr2=0x7fd4a80fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.056+0000 7fd4acd80640 1 -- 172.21.15.119:0/1565175459 shutdown_connections 2024-09-10T18:56:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.056+0000 7fd4acd80640 1 -- 172.21.15.119:0/1565175459 wait complete. 2024-09-10T18:56:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.057+0000 7fd4acd80640 1 Processor -- start 2024-09-10T18:56:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.057+0000 7fd4acd80640 1 -- start start 2024-09-10T18:56:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.057+0000 7fd4acd80640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4a8077480 0x7fd4a81114d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.057+0000 7fd4acd80640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a8111a10 con 0x7fd4a8077480 2024-09-10T18:56:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.057+0000 7fd4a659c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4a8077480 0x7fd4a81114d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.057+0000 7fd4a659c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4a8077480 0x7fd4a81114d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49142/0 (socket says 172.21.15.119:49142) 2024-09-10T18:56:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.057+0000 7fd4a659c640 1 -- 172.21.15.119:0/4075951049 learned_addr learned my addr 172.21.15.119:0/4075951049 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:54.827 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.057+0000 7fd4a659c640 1 -- 172.21.15.119:0/4075951049 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd498009450 con 0x7fd4a8077480 2024-09-10T18:56:54.827 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.057+0000 7fd4a659c640 1 --2- 172.21.15.119:0/4075951049 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4a8077480 0x7fd4a81114d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd49802a7a0 tx=0x7fd4980091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:54.827 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.058+0000 7fd496ffd640 1 -- 172.21.15.119:0/4075951049 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd498032cd0 con 0x7fd4a8077480 2024-09-10T18:56:54.827 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.058+0000 7fd496ffd640 1 -- 172.21.15.119:0/4075951049 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7fd49803c850 con 0x7fd4a8077480 2024-09-10T18:56:54.827 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.058+0000 7fd4acd80640 1 -- 172.21.15.119:0/4075951049 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a8111c10 con 0x7fd4a8077480 2024-09-10T18:56:54.827 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.058+0000 7fd496ffd640 1 -- 172.21.15.119:0/4075951049 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd49803a4f0 con 0x7fd4a8077480 2024-09-10T18:56:54.827 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.058+0000 7fd4acd80640 1 -- 172.21.15.119:0/4075951049 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a8112080 con 0x7fd4a8077480 2024-09-10T18:56:54.827 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.058+0000 7fd496ffd640 1 -- 172.21.15.119:0/4075951049 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 45457+0+0 (secure 0 0 0) 0x7fd49803a760 con 0x7fd4a8077480 2024-09-10T18:56:54.827 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.059+0000 7fd496ffd640 1 --2- 172.21.15.119:0/4075951049 >> v2:172.21.15.119:6800/4184327284 conn(0x7fd490039430 0x7fd49003b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:54.827 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.059+0000 7fd496ffd640 1 -- 172.21.15.119:0/4075951049 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fd49806c850 con 0x7fd4a8077480 2024-09-10T18:56:54.827 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.059+0000 7fd4a5d9b640 1 -- 172.21.15.119:0/4075951049 >> v2:172.21.15.119:6800/4184327284 conn(0x7fd490039430 msgr2=0x7fd49003b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/4184327284 2024-09-10T18:56:54.827 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.059+0000 7fd4a5d9b640 1 --2- 172.21.15.119:0/4075951049 >> v2:172.21.15.119:6800/4184327284 conn(0x7fd490039430 0x7fd49003b8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T18:56:54.827 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.059+0000 7fd4acd80640 1 -- 172.21.15.119:0/4075951049 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4a8112080 con 0x7fd4a8077480 2024-09-10T18:56:54.828 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.063+0000 7fd496ffd640 1 -- 172.21.15.119:0/4075951049 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fd498068080 con 0x7fd4a8077480 2024-09-10T18:56:54.828 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.260+0000 7fd4a5d9b640 1 -- 172.21.15.119:0/4075951049 >> v2:172.21.15.119:6800/4184327284 conn(0x7fd490039430 msgr2=0x7fd49003b8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/4184327284 2024-09-10T18:56:54.828 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.260+0000 7fd4a5d9b640 1 --2- 172.21.15.119:0/4075951049 >> v2:172.21.15.119:6800/4184327284 conn(0x7fd490039430 0x7fd49003b8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T18:56:54.828 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.263+0000 7fd4acd80640 1 -- 172.21.15.119:0/4075951049 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd4a805ed70 con 0x7fd4a8077480 2024-09-10T18:56:54.828 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.264+0000 7fd496ffd640 1 -- 172.21.15.119:0/4075951049 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7fd498037030 con 0x7fd4a8077480 2024-09-10T18:56:54.828 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.266+0000 7fd4acd80640 1 -- 172.21.15.119:0/4075951049 >> v2:172.21.15.119:6800/4184327284 conn(0x7fd490039430 msgr2=0x7fd49003b8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T18:56:54.828 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.266+0000 7fd4acd80640 1 --2- 172.21.15.119:0/4075951049 >> v2:172.21.15.119:6800/4184327284 conn(0x7fd490039430 0x7fd49003b8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:54.828 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.266+0000 7fd4acd80640 1 -- 172.21.15.119:0/4075951049 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4a8077480 msgr2=0x7fd4a81114d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:54.828 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.266+0000 7fd4acd80640 1 --2- 172.21.15.119:0/4075951049 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4a8077480 0x7fd4a81114d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd49802a7a0 tx=0x7fd4980091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:54.828 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.266+0000 7fd4acd80640 1 -- 172.21.15.119:0/4075951049 shutdown_connections 2024-09-10T18:56:54.828 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.266+0000 7fd4acd80640 1 --2- 172.21.15.119:0/4075951049 >> v2:172.21.15.119:6800/4184327284 conn(0x7fd490039430 0x7fd49003b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:54.828 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.266+0000 7fd4acd80640 1 --2- 172.21.15.119:0/4075951049 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4a8077480 0x7fd4a81114d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:54.828 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.266+0000 7fd4acd80640 1 -- 172.21.15.119:0/4075951049 >> 172.21.15.119:0/4075951049 conn(0x7fd4a80fbea0 msgr2=0x7fd4a80fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:54.829 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.266+0000 7fd4acd80640 1 -- 172.21.15.119:0/4075951049 shutdown_connections 2024-09-10T18:56:54.829 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:54.266+0000 7fd4acd80640 1 -- 172.21.15.119:0/4075951049 wait complete. 2024-09-10T18:56:54.829 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for the mgr to restart... 2024-09-10T18:56:54.829 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mgr epoch 5... 2024-09-10T18:56:55.566 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:55.124+0000 7f15cfa5aec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:56:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: Active manager daemon y restarted 2024-09-10T18:56:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: Activating manager daemon y 2024-09-10T18:56:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: osdmap e2: 0 total, 0 up, 0 in 2024-09-10T18:56:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: mgrmap e6: y(active, starting, since 0.00315496s) 2024-09-10T18:56:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:56:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:56:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:56:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:56:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:56:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: Manager daemon y is now available 2024-09-10T18:56:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' 2024-09-10T18:56:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' 2024-09-10T18:56:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:56:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:56:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:56:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:55 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:56:56.439 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-mon[29577]: Found migration_current of "None". Setting to last migration. 2024-09-10T18:56:56.439 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-mon[29577]: mgrmap e7: y(active, since 1.00526s) 2024-09-10T18:56:56.646 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:56:56.647 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-10T18:56:56.647 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-10T18:56:56.647 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:56:56.647 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.313+0000 7f0fc7aef640 1 Processor -- start 2024-09-10T18:56:56.647 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.314+0000 7f0fc7aef640 1 -- start start 2024-09-10T18:56:56.647 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.314+0000 7f0fc7aef640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc0074c60 0x7f0fc0103170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:56.647 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.314+0000 7f0fc7aef640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc01036b0 con 0x7f0fc0074c60 2024-09-10T18:56:56.647 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.314+0000 7f0fc588b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc0074c60 0x7f0fc0103170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:56.647 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.314+0000 7f0fc588b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc0074c60 0x7f0fc0103170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49204/0 (socket says 172.21.15.119:49204) 2024-09-10T18:56:56.647 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.314+0000 7f0fc588b640 1 -- 172.21.15.119:0/1681765500 learned_addr learned my addr 172.21.15.119:0/1681765500 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:56.647 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.314+0000 7f0fc588b640 1 -- 172.21.15.119:0/1681765500 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc01037f0 con 0x7f0fc0074c60 2024-09-10T18:56:56.647 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.315+0000 7f0fc588b640 1 --2- 172.21.15.119:0/1681765500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc0074c60 0x7f0fc0103170 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009840 tx=0x7f0fbc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c81d37cf4c046c97 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:56.648 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.315+0000 7f0fc4889640 1 -- 172.21.15.119:0/1681765500 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fbc02add0 con 0x7f0fc0074c60 2024-09-10T18:56:56.648 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.315+0000 7f0fc4889640 1 -- 172.21.15.119:0/1681765500 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f0fbc032890 con 0x7f0fc0074c60 2024-09-10T18:56:56.648 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.315+0000 7f0fc4889640 1 -- 172.21.15.119:0/1681765500 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fbc0317a0 con 0x7f0fc0074c60 2024-09-10T18:56:56.648 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.316+0000 7f0fc7aef640 1 -- 172.21.15.119:0/1681765500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc0074c60 msgr2=0x7f0fc0103170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:56.648 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.316+0000 7f0fc7aef640 1 --2- 172.21.15.119:0/1681765500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc0074c60 0x7f0fc0103170 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009840 tx=0x7f0fbc0091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:56.648 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.316+0000 7f0fc7aef640 1 -- 172.21.15.119:0/1681765500 shutdown_connections 2024-09-10T18:56:56.648 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.316+0000 7f0fc7aef640 1 --2- 172.21.15.119:0/1681765500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc0074c60 0x7f0fc0103170 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:56.648 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.316+0000 7f0fc7aef640 1 -- 172.21.15.119:0/1681765500 >> 172.21.15.119:0/1681765500 conn(0x7f0fc00fabc0 msgr2=0x7f0fc00fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:56.648 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.316+0000 7f0fc7aef640 1 -- 172.21.15.119:0/1681765500 shutdown_connections 2024-09-10T18:56:56.648 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.316+0000 7f0fc7aef640 1 -- 172.21.15.119:0/1681765500 wait complete. 2024-09-10T18:56:56.648 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.316+0000 7f0fc7aef640 1 Processor -- start 2024-09-10T18:56:56.648 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.317+0000 7f0fc7aef640 1 -- start start 2024-09-10T18:56:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.317+0000 7f0fc7aef640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc0074c60 0x7f0fc010f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.317+0000 7f0fc7aef640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc010f830 con 0x7f0fc0074c60 2024-09-10T18:56:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.317+0000 7f0fc588b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc0074c60 0x7f0fc010f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.317+0000 7f0fc588b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc0074c60 0x7f0fc010f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49218/0 (socket says 172.21.15.119:49218) 2024-09-10T18:56:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.317+0000 7f0fc588b640 1 -- 172.21.15.119:0/370081472 learned_addr learned my addr 172.21.15.119:0/370081472 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.317+0000 7f0fc588b640 1 -- 172.21.15.119:0/370081472 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc009450 con 0x7f0fc0074c60 2024-09-10T18:56:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.317+0000 7f0fc588b640 1 --2- 172.21.15.119:0/370081472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc0074c60 0x7f0fc010f2f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0313e0 tx=0x7f0fbc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.318+0000 7f0fb67fc640 1 -- 172.21.15.119:0/370081472 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fbc032cd0 con 0x7f0fc0074c60 2024-09-10T18:56:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.318+0000 7f0fb67fc640 1 -- 172.21.15.119:0/370081472 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f0fbc03c850 con 0x7f0fc0074c60 2024-09-10T18:56:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.318+0000 7f0fc7aef640 1 -- 172.21.15.119:0/370081472 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc010fa30 con 0x7f0fc0074c60 2024-09-10T18:56:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.318+0000 7f0fb67fc640 1 -- 172.21.15.119:0/370081472 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0fbc03a3b0 con 0x7f0fc0074c60 2024-09-10T18:56:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.318+0000 7f0fc7aef640 1 -- 172.21.15.119:0/370081472 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc010fea0 con 0x7f0fc0074c60 2024-09-10T18:56:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.319+0000 7f0fb67fc640 1 -- 172.21.15.119:0/370081472 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 6) v1 ==== 45259+0+0 (secure 0 0 0) 0x7f0fbc03a620 con 0x7f0fc0074c60 2024-09-10T18:56:56.649 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:55.319+0000 7f0fb67fc640 1 -- 172.21.15.119:0/370081472 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f0fbc06cff0 con 0x7f0fc0074c60 2024-09-10T18:56:56.650 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.129+0000 7f0fb67fc640 1 -- 172.21.15.119:0/370081472 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mgrmap(e 7) v1 ==== 45351+0+0 (secure 0 0 0) 0x7f0fbc06c7c0 con 0x7f0fc0074c60 2024-09-10T18:56:56.650 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.129+0000 7f0fb67fc640 1 --2- 172.21.15.119:0/370081472 >> v2:172.21.15.119:6800/1136125279 conn(0x7f0fac039f70 0x7f0fac03c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:56.650 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.129+0000 7f0fb67fc640 1 -- 172.21.15.119:0/370081472 --> v2:172.21.15.119:6800/1136125279 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0fac03c890 con 0x7f0fac039f70 2024-09-10T18:56:56.650 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.129+0000 7f0fc508a640 1 --2- 172.21.15.119:0/370081472 >> v2:172.21.15.119:6800/1136125279 conn(0x7f0fac039f70 0x7f0fac03c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:56.650 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.130+0000 7f0fc508a640 1 --2- 172.21.15.119:0/370081472 >> v2:172.21.15.119:6800/1136125279 conn(0x7f0fac039f70 0x7f0fac03c350 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0006fd0 tx=0x7f0fb0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:56.650 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.130+0000 7f0fb67fc640 1 -- 172.21.15.119:0/370081472 <== mgr.14118 v2:172.21.15.119:6800/1136125279 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f0fac03c890 con 0x7f0fac039f70 2024-09-10T18:56:56.650 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.135+0000 7f0fc7aef640 1 -- 172.21.15.119:0/370081472 --> v2:172.21.15.119:6800/1136125279 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f0fc005ed70 con 0x7f0fac039f70 2024-09-10T18:56:56.650 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.135+0000 7f0fb67fc640 1 -- 172.21.15.119:0/370081472 <== mgr.14118 v2:172.21.15.119:6800/1136125279 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f0fc005ed70 con 0x7f0fac039f70 2024-09-10T18:56:56.650 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.136+0000 7f0fc7aef640 1 -- 172.21.15.119:0/370081472 >> v2:172.21.15.119:6800/1136125279 conn(0x7f0fac039f70 msgr2=0x7f0fac03c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:56.650 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.136+0000 7f0fc7aef640 1 --2- 172.21.15.119:0/370081472 >> v2:172.21.15.119:6800/1136125279 conn(0x7f0fac039f70 0x7f0fac03c350 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0fb0006fd0 tx=0x7f0fb0006db0 comp rx=0 tx=0).stop 2024-09-10T18:56:56.650 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.136+0000 7f0fc7aef640 1 -- 172.21.15.119:0/370081472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc0074c60 msgr2=0x7f0fc010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:56.650 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.136+0000 7f0fc7aef640 1 --2- 172.21.15.119:0/370081472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc0074c60 0x7f0fc010f2f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc0313e0 tx=0x7f0fbc0091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:56.650 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.136+0000 7f0fc7aef640 1 -- 172.21.15.119:0/370081472 shutdown_connections 2024-09-10T18:56:56.651 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.136+0000 7f0fc7aef640 1 --2- 172.21.15.119:0/370081472 >> v2:172.21.15.119:6800/1136125279 conn(0x7f0fac039f70 0x7f0fac03c350 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:56.651 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.136+0000 7f0fc7aef640 1 --2- 172.21.15.119:0/370081472 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc0074c60 0x7f0fc010f2f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:56.651 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.136+0000 7f0fc7aef640 1 -- 172.21.15.119:0/370081472 >> 172.21.15.119:0/370081472 conn(0x7f0fc00fabc0 msgr2=0x7f0fc00fb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:56.651 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.136+0000 7f0fc7aef640 1 -- 172.21.15.119:0/370081472 shutdown_connections 2024-09-10T18:56:56.651 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:56.136+0000 7f0fc7aef640 1 -- 172.21.15.119:0/370081472 wait complete. 2024-09-10T18:56:56.651 INFO:teuthology.orchestra.run.smithi119.stdout:mgr epoch 5 is available 2024-09-10T18:56:56.651 INFO:teuthology.orchestra.run.smithi119.stdout:Setting orchestrator backend to cephadm... 2024-09-10T18:56:57.017 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:56.877+0000 7f1581e48640 -1 log_channel(cephadm) log [ERR] : [10/Sep/2024:18:56:56] ENGINE Error in HTTPServer.serve 2024-09-10T18:56:57.017 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: Traceback (most recent call last): 2024-09-10T18:56:57.018 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:56:57.018 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: self._connections.run(self.expiration_interval) 2024-09-10T18:56:57.018 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:56:57.018 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: self._run(expiration_interval) 2024-09-10T18:56:57.018 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:56:57.018 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:56:57.018 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:56:57.018 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:56:57.018 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:56:57.018 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: s = self.context.wrap_socket( 2024-09-10T18:56:57.018 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:56:57.018 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: return self.sslsocket_class._create( 2024-09-10T18:56:57.018 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:56:57.019 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: self.do_handshake() 2024-09-10T18:56:57.019 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:56:57.019 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: self._sslobj.do_handshake() 2024-09-10T18:56:57.019 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:56:57.019 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:56 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:56:57.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:57 smithi119 ceph-mon[29577]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-10T18:56:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:57 smithi119 ceph-mon[29577]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-10T18:56:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:57 smithi119 ceph-mon[29577]: [10/Sep/2024:18:56:56] ENGINE Bus STARTING 2024-09-10T18:56:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:57 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' 2024-09-10T18:56:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:57 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:56:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:57 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' 2024-09-10T18:56:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:57 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:56:57.836 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.091+0000 7f5321bee640 1 Processor -- start 2024-09-10T18:56:57.836 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.091+0000 7f5321bee640 1 -- start start 2024-09-10T18:56:57.837 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.092+0000 7f5321bee640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f531c0ff4a0 0x7f531c0ff8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:57.837 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.092+0000 7f5321bee640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c0ffdf0 con 0x7f531c0ff4a0 2024-09-10T18:56:57.837 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.092+0000 7f531b7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f531c0ff4a0 0x7f531c0ff8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:57.837 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.092+0000 7f531b7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f531c0ff4a0 0x7f531c0ff8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49234/0 (socket says 172.21.15.119:49234) 2024-09-10T18:56:57.837 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.092+0000 7f531b7fe640 1 -- 172.21.15.119:0/3428373067 learned_addr learned my addr 172.21.15.119:0/3428373067 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:57.837 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.092+0000 7f531b7fe640 1 -- 172.21.15.119:0/3428373067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531c0fff30 con 0x7f531c0ff4a0 2024-09-10T18:56:57.837 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.093+0000 7f531b7fe640 1 --2- 172.21.15.119:0/3428373067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f531c0ff4a0 0x7f531c0ff8b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f530c00baf0 tx=0x7f530c00b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d3c44ef6ae98af server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:57.837 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.093+0000 7f531affd640 1 -- 172.21.15.119:0/3428373067 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f530c032640 con 0x7f531c0ff4a0 2024-09-10T18:56:57.837 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.093+0000 7f531affd640 1 -- 172.21.15.119:0/3428373067 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f530c032bf0 con 0x7f531c0ff4a0 2024-09-10T18:56:57.837 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.093+0000 7f531affd640 1 -- 172.21.15.119:0/3428373067 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f530c031b50 con 0x7f531c0ff4a0 2024-09-10T18:56:57.837 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.093+0000 7f5321bee640 1 -- 172.21.15.119:0/3428373067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f531c0ff4a0 msgr2=0x7f531c0ff8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:57.837 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.093+0000 7f5321bee640 1 --2- 172.21.15.119:0/3428373067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f531c0ff4a0 0x7f531c0ff8b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f530c00baf0 tx=0x7f530c00b460 comp rx=0 tx=0).stop 2024-09-10T18:56:57.837 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.093+0000 7f5321bee640 1 -- 172.21.15.119:0/3428373067 shutdown_connections 2024-09-10T18:56:57.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.093+0000 7f5321bee640 1 --2- 172.21.15.119:0/3428373067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f531c0ff4a0 0x7f531c0ff8b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:57.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.093+0000 7f5321bee640 1 -- 172.21.15.119:0/3428373067 >> 172.21.15.119:0/3428373067 conn(0x7f531c0faca0 msgr2=0x7f531c0fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:57.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.094+0000 7f5321bee640 1 -- 172.21.15.119:0/3428373067 shutdown_connections 2024-09-10T18:56:57.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.094+0000 7f5321bee640 1 -- 172.21.15.119:0/3428373067 wait complete. 2024-09-10T18:56:57.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.094+0000 7f5321bee640 1 Processor -- start 2024-09-10T18:56:57.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.094+0000 7f5321bee640 1 -- start start 2024-09-10T18:56:57.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.094+0000 7f5321bee640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f531c0ff4a0 0x7f531c10d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:57.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.094+0000 7f5321bee640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531c10d5f0 con 0x7f531c0ff4a0 2024-09-10T18:56:57.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.094+0000 7f531b7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f531c0ff4a0 0x7f531c10d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:57.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.094+0000 7f531b7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f531c0ff4a0 0x7f531c10d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49236/0 (socket says 172.21.15.119:49236) 2024-09-10T18:56:57.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.094+0000 7f531b7fe640 1 -- 172.21.15.119:0/4021996249 learned_addr learned my addr 172.21.15.119:0/4021996249 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:57.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.095+0000 7f531b7fe640 1 -- 172.21.15.119:0/4021996249 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f530c00b700 con 0x7f531c0ff4a0 2024-09-10T18:56:57.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.095+0000 7f531b7fe640 1 --2- 172.21.15.119:0/4021996249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f531c0ff4a0 0x7f531c10d0b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f530c00b2c0 tx=0x7f530c00b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:57.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.095+0000 7f5318ff9640 1 -- 172.21.15.119:0/4021996249 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f530c031bf0 con 0x7f531c0ff4a0 2024-09-10T18:56:57.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.095+0000 7f5318ff9640 1 -- 172.21.15.119:0/4021996249 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f530c032850 con 0x7f531c0ff4a0 2024-09-10T18:56:57.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.095+0000 7f5321bee640 1 -- 172.21.15.119:0/4021996249 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531c10d7f0 con 0x7f531c0ff4a0 2024-09-10T18:56:57.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.095+0000 7f5318ff9640 1 -- 172.21.15.119:0/4021996249 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f530c0463f0 con 0x7f531c0ff4a0 2024-09-10T18:56:57.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.095+0000 7f5321bee640 1 -- 172.21.15.119:0/4021996249 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531c10dc00 con 0x7f531c0ff4a0 2024-09-10T18:56:57.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.096+0000 7f5318ff9640 1 -- 172.21.15.119:0/4021996249 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 7) v1 ==== 45351+0+0 (secure 0 0 0) 0x7f530c03b420 con 0x7f531c0ff4a0 2024-09-10T18:56:57.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.096+0000 7f5318ff9640 1 --2- 172.21.15.119:0/4021996249 >> v2:172.21.15.119:6800/1136125279 conn(0x7f5308039360 0x7f530803b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:57.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.096+0000 7f5321bee640 1 -- 172.21.15.119:0/4021996249 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f531c04b720 con 0x7f531c0ff4a0 2024-09-10T18:56:57.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.096+0000 7f5318ff9640 1 -- 172.21.15.119:0/4021996249 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f530c06f1b0 con 0x7f531c0ff4a0 2024-09-10T18:56:57.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.096+0000 7f5312ffd640 1 --2- 172.21.15.119:0/4021996249 >> v2:172.21.15.119:6800/1136125279 conn(0x7f5308039360 0x7f530803b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:57.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.096+0000 7f5312ffd640 1 --2- 172.21.15.119:0/4021996249 >> v2:172.21.15.119:6800/1136125279 conn(0x7f5308039360 0x7f530803b810 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5304006fd0 tx=0x7f5304006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:57.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.101+0000 7f5318ff9640 1 -- 172.21.15.119:0/4021996249 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f530c036070 con 0x7f531c0ff4a0 2024-09-10T18:56:57.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.265+0000 7f5321bee640 1 -- 172.21.15.119:0/4021996249 --> v2:172.21.15.119:6800/1136125279 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f531c10e5e0 con 0x7f5308039360 2024-09-10T18:56:57.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.273+0000 7f5318ff9640 1 -- 172.21.15.119:0/4021996249 <== mgr.14118 v2:172.21.15.119:6800/1136125279 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f531c10e5e0 con 0x7f5308039360 2024-09-10T18:56:57.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.275+0000 7f5321bee640 1 -- 172.21.15.119:0/4021996249 >> v2:172.21.15.119:6800/1136125279 conn(0x7f5308039360 msgr2=0x7f530803b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:57.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.275+0000 7f5321bee640 1 --2- 172.21.15.119:0/4021996249 >> v2:172.21.15.119:6800/1136125279 conn(0x7f5308039360 0x7f530803b810 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5304006fd0 tx=0x7f5304006db0 comp rx=0 tx=0).stop 2024-09-10T18:56:57.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.275+0000 7f5321bee640 1 -- 172.21.15.119:0/4021996249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f531c0ff4a0 msgr2=0x7f531c10d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:57.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.275+0000 7f5321bee640 1 --2- 172.21.15.119:0/4021996249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f531c0ff4a0 0x7f531c10d0b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f530c00b2c0 tx=0x7f530c00b460 comp rx=0 tx=0).stop 2024-09-10T18:56:57.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.275+0000 7f5321bee640 1 -- 172.21.15.119:0/4021996249 shutdown_connections 2024-09-10T18:56:57.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.275+0000 7f5321bee640 1 --2- 172.21.15.119:0/4021996249 >> v2:172.21.15.119:6800/1136125279 conn(0x7f5308039360 0x7f530803b810 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:57.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.275+0000 7f5321bee640 1 --2- 172.21.15.119:0/4021996249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f531c0ff4a0 0x7f531c10d0b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:57.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.275+0000 7f5321bee640 1 -- 172.21.15.119:0/4021996249 >> 172.21.15.119:0/4021996249 conn(0x7f531c0faca0 msgr2=0x7f531c103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:57.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.275+0000 7f5321bee640 1 -- 172.21.15.119:0/4021996249 shutdown_connections 2024-09-10T18:56:57.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:57.275+0000 7f5321bee640 1 -- 172.21.15.119:0/4021996249 wait complete. 2024-09-10T18:56:58.423 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:58 smithi119 ceph-mon[29577]: [10/Sep/2024:18:56:56] ENGINE Serving on https://172.21.15.119:7150 2024-09-10T18:56:58.424 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:58 smithi119 ceph-mon[29577]: [10/Sep/2024:18:56:56] ENGINE Bus STARTED 2024-09-10T18:56:58.424 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:58 smithi119 ceph-mon[29577]: [10/Sep/2024:18:56:56] ENGINE Error in HTTPServer.serve 2024-09-10T18:56:58.424 INFO:journalctl@ceph.mon.a.smithi119.stdout: Traceback (most recent call last): 2024-09-10T18:56:58.424 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:56:58.424 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._connections.run(self.expiration_interval) 2024-09-10T18:56:58.424 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:56:58.424 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._run(expiration_interval) 2024-09-10T18:56:58.424 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:56:58.424 INFO:journalctl@ceph.mon.a.smithi119.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:56:58.424 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:56:58.424 INFO:journalctl@ceph.mon.a.smithi119.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:56:58.425 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:56:58.425 INFO:journalctl@ceph.mon.a.smithi119.stdout: s = self.context.wrap_socket( 2024-09-10T18:56:58.425 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:56:58.425 INFO:journalctl@ceph.mon.a.smithi119.stdout: return self.sslsocket_class._create( 2024-09-10T18:56:58.425 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:56:58.425 INFO:journalctl@ceph.mon.a.smithi119.stdout: self.do_handshake() 2024-09-10T18:56:58.425 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:56:58.425 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._sslobj.do_handshake() 2024-09-10T18:56:58.425 INFO:journalctl@ceph.mon.a.smithi119.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:56:58.425 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:58 smithi119 ceph-mon[29577]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:56:58.425 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:58 smithi119 ceph-mon[29577]: mgrmap e8: y(active, since 2s) 2024-09-10T18:56:59.051 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-10T18:56:59.052 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.250+0000 7f355741d640 1 Processor -- start 2024-09-10T18:56:59.052 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.250+0000 7f355741d640 1 -- start start 2024-09-10T18:56:59.052 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.250+0000 7f355741d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3550075460 0x7f3550103210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:59.052 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.250+0000 7f355741d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3550103750 con 0x7f3550075460 2024-09-10T18:56:59.052 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.250+0000 7f35551b9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3550075460 0x7f3550103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:59.052 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.250+0000 7f35551b9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3550075460 0x7f3550103210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49242/0 (socket says 172.21.15.119:49242) 2024-09-10T18:56:59.052 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.250+0000 7f35551b9640 1 -- 172.21.15.119:0/3798935835 learned_addr learned my addr 172.21.15.119:0/3798935835 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:59.052 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.250+0000 7f35551b9640 1 -- 172.21.15.119:0/3798935835 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3550103890 con 0x7f3550075460 2024-09-10T18:56:59.052 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.251+0000 7f35551b9640 1 --2- 172.21.15.119:0/3798935835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3550075460 0x7f3550103210 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f354c009840 tx=0x7f354c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7491063fe8545d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:59.052 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.251+0000 7f3547fff640 1 -- 172.21.15.119:0/3798935835 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f354c02add0 con 0x7f3550075460 2024-09-10T18:56:59.052 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.251+0000 7f3547fff640 1 -- 172.21.15.119:0/3798935835 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f354c032890 con 0x7f3550075460 2024-09-10T18:56:59.053 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.251+0000 7f3547fff640 1 -- 172.21.15.119:0/3798935835 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f354c0317a0 con 0x7f3550075460 2024-09-10T18:56:59.053 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.252+0000 7f355741d640 1 -- 172.21.15.119:0/3798935835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3550075460 msgr2=0x7f3550103210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:59.053 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.252+0000 7f355741d640 1 --2- 172.21.15.119:0/3798935835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3550075460 0x7f3550103210 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f354c009840 tx=0x7f354c0091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:59.053 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.252+0000 7f355741d640 1 -- 172.21.15.119:0/3798935835 shutdown_connections 2024-09-10T18:56:59.053 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.252+0000 7f355741d640 1 --2- 172.21.15.119:0/3798935835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3550075460 0x7f3550103210 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:59.053 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.252+0000 7f355741d640 1 -- 172.21.15.119:0/3798935835 >> 172.21.15.119:0/3798935835 conn(0x7f35500fac60 msgr2=0x7f35500fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:59.053 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.252+0000 7f355741d640 1 -- 172.21.15.119:0/3798935835 shutdown_connections 2024-09-10T18:56:59.053 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.252+0000 7f355741d640 1 -- 172.21.15.119:0/3798935835 wait complete. 2024-09-10T18:56:59.053 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.253+0000 7f355741d640 1 Processor -- start 2024-09-10T18:56:59.053 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.253+0000 7f355741d640 1 -- start start 2024-09-10T18:56:59.053 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.253+0000 7f355741d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3550075460 0x7f355010f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:59.053 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.253+0000 7f355741d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355010f850 con 0x7f3550075460 2024-09-10T18:56:59.054 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.253+0000 7f35551b9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3550075460 0x7f355010f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:59.054 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.253+0000 7f35551b9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3550075460 0x7f355010f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49248/0 (socket says 172.21.15.119:49248) 2024-09-10T18:56:59.054 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.253+0000 7f35551b9640 1 -- 172.21.15.119:0/674729347 learned_addr learned my addr 172.21.15.119:0/674729347 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:56:59.054 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.253+0000 7f35551b9640 1 -- 172.21.15.119:0/674729347 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354c009450 con 0x7f3550075460 2024-09-10T18:56:59.054 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.253+0000 7f35551b9640 1 --2- 172.21.15.119:0/674729347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3550075460 0x7f355010f310 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f354c0313e0 tx=0x7f354c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:59.054 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.253+0000 7f3545ffb640 1 -- 172.21.15.119:0/674729347 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f354c031e70 con 0x7f3550075460 2024-09-10T18:56:59.054 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.253+0000 7f3545ffb640 1 -- 172.21.15.119:0/674729347 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f354c03c850 con 0x7f3550075460 2024-09-10T18:56:59.054 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.253+0000 7f355741d640 1 -- 172.21.15.119:0/674729347 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355010fa50 con 0x7f3550075460 2024-09-10T18:56:59.054 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.253+0000 7f3545ffb640 1 -- 172.21.15.119:0/674729347 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f354c03a580 con 0x7f3550075460 2024-09-10T18:56:59.054 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.254+0000 7f355741d640 1 -- 172.21.15.119:0/674729347 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355010fec0 con 0x7f3550075460 2024-09-10T18:56:59.054 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.254+0000 7f3545ffb640 1 -- 172.21.15.119:0/674729347 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 45457+0+0 (secure 0 0 0) 0x7f354c03c400 con 0x7f3550075460 2024-09-10T18:56:59.054 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.255+0000 7f355741d640 1 -- 172.21.15.119:0/674729347 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355004b720 con 0x7f3550075460 2024-09-10T18:56:59.055 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.255+0000 7f3545ffb640 1 --2- 172.21.15.119:0/674729347 >> v2:172.21.15.119:6800/1136125279 conn(0x7f353c039470 0x7f353c03b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:56:59.055 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.255+0000 7f3545ffb640 1 -- 172.21.15.119:0/674729347 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f354c06c690 con 0x7f3550075460 2024-09-10T18:56:59.055 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.255+0000 7f35549b8640 1 --2- 172.21.15.119:0/674729347 >> v2:172.21.15.119:6800/1136125279 conn(0x7f353c039470 0x7f353c03b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:56:59.055 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.255+0000 7f35549b8640 1 --2- 172.21.15.119:0/674729347 >> v2:172.21.15.119:6800/1136125279 conn(0x7f353c039470 0x7f353c03b920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3540006fd0 tx=0x7f3540006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:56:59.055 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.259+0000 7f3545ffb640 1 -- 172.21.15.119:0/674729347 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f354c03c6b0 con 0x7f3550075460 2024-09-10T18:56:59.055 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.420+0000 7f355741d640 1 -- 172.21.15.119:0/674729347 --> v2:172.21.15.119:6800/1136125279 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f355010fec0 con 0x7f353c039470 2024-09-10T18:56:59.055 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.421+0000 7f3545ffb640 1 -- 172.21.15.119:0/674729347 <== mgr.14118 v2:172.21.15.119:6800/1136125279 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f355010fec0 con 0x7f353c039470 2024-09-10T18:56:59.055 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.423+0000 7f355741d640 1 -- 172.21.15.119:0/674729347 >> v2:172.21.15.119:6800/1136125279 conn(0x7f353c039470 msgr2=0x7f353c03b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:59.055 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.423+0000 7f355741d640 1 --2- 172.21.15.119:0/674729347 >> v2:172.21.15.119:6800/1136125279 conn(0x7f353c039470 0x7f353c03b920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3540006fd0 tx=0x7f3540006db0 comp rx=0 tx=0).stop 2024-09-10T18:56:59.055 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.423+0000 7f355741d640 1 -- 172.21.15.119:0/674729347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3550075460 msgr2=0x7f355010f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:56:59.055 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.423+0000 7f355741d640 1 --2- 172.21.15.119:0/674729347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3550075460 0x7f355010f310 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f354c0313e0 tx=0x7f354c0091d0 comp rx=0 tx=0).stop 2024-09-10T18:56:59.056 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.423+0000 7f355741d640 1 -- 172.21.15.119:0/674729347 shutdown_connections 2024-09-10T18:56:59.056 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.423+0000 7f355741d640 1 --2- 172.21.15.119:0/674729347 >> v2:172.21.15.119:6800/1136125279 conn(0x7f353c039470 0x7f353c03b920 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:59.056 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.423+0000 7f355741d640 1 --2- 172.21.15.119:0/674729347 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3550075460 0x7f355010f310 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:56:59.056 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.423+0000 7f355741d640 1 -- 172.21.15.119:0/674729347 >> 172.21.15.119:0/674729347 conn(0x7f35500fac60 msgr2=0x7f35500fb850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:56:59.056 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.423+0000 7f355741d640 1 -- 172.21.15.119:0/674729347 shutdown_connections 2024-09-10T18:56:59.056 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:58.423+0000 7f355741d640 1 -- 172.21.15.119:0/674729347 wait complete. 2024-09-10T18:56:59.056 INFO:teuthology.orchestra.run.smithi119.stdout:Generating ssh key... 2024-09-10T18:56:59.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-mon[29577]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:57:00.258 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: Generating public/private rsa key pair. 2024-09-10T18:57:00.258 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: Your identification has been saved in /tmp/tmpaft6kw2l/key 2024-09-10T18:57:00.258 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: Your public key has been saved in /tmp/tmpaft6kw2l/key.pub 2024-09-10T18:57:00.259 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: The key fingerprint is: 2024-09-10T18:57:00.259 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: SHA256:bVI/Ply+c2sCPKFTwl440cs5e1qj+yDk3r7QbtT/piw ceph-3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:57:00.259 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: The key's randomart image is: 2024-09-10T18:57:00.259 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: +---[RSA 3072]----+ 2024-09-10T18:57:00.259 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: | . | 2024-09-10T18:57:00.259 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: | . . | 2024-09-10T18:57:00.259 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: | . = o | 2024-09-10T18:57:00.259 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: | B X | 2024-09-10T18:57:00.259 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: | S.X B . | 2024-09-10T18:57:00.259 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: | o*.O O | 2024-09-10T18:57:00.259 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: | +oo@ + | 2024-09-10T18:57:00.259 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: | . =+E+.+o| 2024-09-10T18:57:00.259 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: | .o*+o**+| 2024-09-10T18:57:00.259 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:56:59 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: +----[SHA256]-----+ 2024-09-10T18:57:00.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:00 smithi119 ceph-mon[29577]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:57:00.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:00 smithi119 ceph-mon[29577]: Generating ssh key... 2024-09-10T18:57:00.517 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:00 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' 2024-09-10T18:57:00.518 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:00 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' 2024-09-10T18:57:00.524 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.579+0000 7f371ba69640 1 Processor -- start 2024-09-10T18:57:00.524 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.579+0000 7f371ba69640 1 -- start start 2024-09-10T18:57:00.525 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.580+0000 7f371ba69640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37140ff440 0x7f37140ff850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:00.525 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.580+0000 7f371ba69640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37140ffd90 con 0x7f37140ff440 2024-09-10T18:57:00.525 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.580+0000 7f3719805640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37140ff440 0x7f37140ff850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:00.525 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.580+0000 7f3719805640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37140ff440 0x7f37140ff850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49256/0 (socket says 172.21.15.119:49256) 2024-09-10T18:57:00.525 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.580+0000 7f3719805640 1 -- 172.21.15.119:0/846136307 learned_addr learned my addr 172.21.15.119:0/846136307 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:00.525 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.580+0000 7f3719805640 1 -- 172.21.15.119:0/846136307 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37140ffed0 con 0x7f37140ff440 2024-09-10T18:57:00.525 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.581+0000 7f3719805640 1 --2- 172.21.15.119:0/846136307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37140ff440 0x7f37140ff850 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3704009840 tx=0x7f37040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5175eafee407b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:00.525 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.581+0000 7f3718803640 1 -- 172.21.15.119:0/846136307 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f370402add0 con 0x7f37140ff440 2024-09-10T18:57:00.525 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.581+0000 7f3718803640 1 -- 172.21.15.119:0/846136307 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f3704032890 con 0x7f37140ff440 2024-09-10T18:57:00.525 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.581+0000 7f3718803640 1 -- 172.21.15.119:0/846136307 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f37040317a0 con 0x7f37140ff440 2024-09-10T18:57:00.525 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.581+0000 7f371ba69640 1 -- 172.21.15.119:0/846136307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37140ff440 msgr2=0x7f37140ff850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:00.525 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.581+0000 7f371ba69640 1 --2- 172.21.15.119:0/846136307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37140ff440 0x7f37140ff850 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3704009840 tx=0x7f37040091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:00.525 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.581+0000 7f371ba69640 1 -- 172.21.15.119:0/846136307 shutdown_connections 2024-09-10T18:57:00.526 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.581+0000 7f371ba69640 1 --2- 172.21.15.119:0/846136307 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37140ff440 0x7f37140ff850 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:00.526 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.581+0000 7f371ba69640 1 -- 172.21.15.119:0/846136307 >> 172.21.15.119:0/846136307 conn(0x7f37140fabc0 msgr2=0x7f37140fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:00.526 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.582+0000 7f371ba69640 1 -- 172.21.15.119:0/846136307 shutdown_connections 2024-09-10T18:57:00.526 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.582+0000 7f371ba69640 1 -- 172.21.15.119:0/846136307 wait complete. 2024-09-10T18:57:00.526 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.582+0000 7f371ba69640 1 Processor -- start 2024-09-10T18:57:00.526 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.582+0000 7f371ba69640 1 -- start start 2024-09-10T18:57:00.526 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.582+0000 7f371ba69640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37140ff440 0x7f3714074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:00.526 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.582+0000 7f371ba69640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3714111fb0 con 0x7f37140ff440 2024-09-10T18:57:00.526 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.582+0000 7f3719805640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37140ff440 0x7f3714074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:00.527 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.582+0000 7f3719805640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37140ff440 0x7f3714074b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49258/0 (socket says 172.21.15.119:49258) 2024-09-10T18:57:00.527 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.582+0000 7f3719805640 1 -- 172.21.15.119:0/2825223760 learned_addr learned my addr 172.21.15.119:0/2825223760 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:00.527 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.583+0000 7f3719805640 1 -- 172.21.15.119:0/2825223760 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3704009450 con 0x7f37140ff440 2024-09-10T18:57:00.527 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.583+0000 7f3719805640 1 --2- 172.21.15.119:0/2825223760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37140ff440 0x7f3714074b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f37040314e0 tx=0x7f37040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:00.527 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.583+0000 7f370a7fc640 1 -- 172.21.15.119:0/2825223760 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f370403c400 con 0x7f37140ff440 2024-09-10T18:57:00.527 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.583+0000 7f370a7fc640 1 -- 172.21.15.119:0/2825223760 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f370403c9b0 con 0x7f37140ff440 2024-09-10T18:57:00.527 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.583+0000 7f370a7fc640 1 -- 172.21.15.119:0/2825223760 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f370403a530 con 0x7f37140ff440 2024-09-10T18:57:00.527 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.583+0000 7f371ba69640 1 -- 172.21.15.119:0/2825223760 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37141120f0 con 0x7f37140ff440 2024-09-10T18:57:00.527 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.583+0000 7f371ba69640 1 -- 172.21.15.119:0/2825223760 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3714112560 con 0x7f37140ff440 2024-09-10T18:57:00.527 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.584+0000 7f370a7fc640 1 -- 172.21.15.119:0/2825223760 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 45457+0+0 (secure 0 0 0) 0x7f370403c560 con 0x7f37140ff440 2024-09-10T18:57:00.527 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.584+0000 7f371ba69640 1 -- 172.21.15.119:0/2825223760 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f371404b720 con 0x7f37140ff440 2024-09-10T18:57:00.527 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.584+0000 7f370a7fc640 1 --2- 172.21.15.119:0/2825223760 >> v2:172.21.15.119:6800/1136125279 conn(0x7f3700039470 0x7f370003b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:00.528 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.584+0000 7f370a7fc640 1 -- 172.21.15.119:0/2825223760 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f370406c780 con 0x7f37140ff440 2024-09-10T18:57:00.528 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.584+0000 7f3719004640 1 --2- 172.21.15.119:0/2825223760 >> v2:172.21.15.119:6800/1136125279 conn(0x7f3700039470 0x7f370003b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:00.528 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.585+0000 7f3719004640 1 --2- 172.21.15.119:0/2825223760 >> v2:172.21.15.119:6800/1136125279 conn(0x7f3700039470 0x7f370003b920 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3710006fd0 tx=0x7f3710006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:00.528 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.588+0000 7f370a7fc640 1 -- 172.21.15.119:0/2825223760 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f370403b360 con 0x7f37140ff440 2024-09-10T18:57:00.528 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.743+0000 7f371ba69640 1 -- 172.21.15.119:0/2825223760 --> v2:172.21.15.119:6800/1136125279 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f37140fbe40 con 0x7f3700039470 2024-09-10T18:57:00.528 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.962+0000 7f370a7fc640 1 -- 172.21.15.119:0/2825223760 <== mgr.14118 v2:172.21.15.119:6800/1136125279 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f37140fbe40 con 0x7f3700039470 2024-09-10T18:57:00.528 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.964+0000 7f371ba69640 1 -- 172.21.15.119:0/2825223760 >> v2:172.21.15.119:6800/1136125279 conn(0x7f3700039470 msgr2=0x7f370003b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:00.528 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.964+0000 7f371ba69640 1 --2- 172.21.15.119:0/2825223760 >> v2:172.21.15.119:6800/1136125279 conn(0x7f3700039470 0x7f370003b920 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3710006fd0 tx=0x7f3710006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:00.528 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.964+0000 7f371ba69640 1 -- 172.21.15.119:0/2825223760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37140ff440 msgr2=0x7f3714074b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:00.529 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.964+0000 7f371ba69640 1 --2- 172.21.15.119:0/2825223760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37140ff440 0x7f3714074b20 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f37040314e0 tx=0x7f37040091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:00.529 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.964+0000 7f371ba69640 1 -- 172.21.15.119:0/2825223760 shutdown_connections 2024-09-10T18:57:00.529 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.964+0000 7f371ba69640 1 --2- 172.21.15.119:0/2825223760 >> v2:172.21.15.119:6800/1136125279 conn(0x7f3700039470 0x7f370003b920 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:00.529 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.964+0000 7f371ba69640 1 --2- 172.21.15.119:0/2825223760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37140ff440 0x7f3714074b20 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:00.529 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.964+0000 7f371ba69640 1 -- 172.21.15.119:0/2825223760 >> 172.21.15.119:0/2825223760 conn(0x7f37140fabc0 msgr2=0x7f37140fb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:00.529 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.964+0000 7f371ba69640 1 -- 172.21.15.119:0/2825223760 shutdown_connections 2024-09-10T18:57:00.529 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:56:59.964+0000 7f371ba69640 1 -- 172.21.15.119:0/2825223760 wait complete. 2024-09-10T18:57:01.698 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2N27W4TumTIc76bjdOh9eW79+HQvmxaekWa83O2GPrBJax8C3Nf5Halpxkx/hAUF4oiY3TDJYWoNcfz+x9U6o4RX7VBl0/W7Puv2OLsbK2NDwwLtpqA7SReYCxd2hwoBkiptVPp7N6twHr5uMyStlPvWlQjyO+WiTsOauYzA5nTvX/mDecNQ3920klrymbLtNGbBJiXACX/okQ42JpqmIuvjm/SSGFx7yZcSREQK5/MdoFbv3vWT3p80UoFAtJMXqyQ+Lw1uvW0j7jbff2k4tcBK35UYjhmBCAJ5atEFuQc+SEVAkBi4z6/M1u3W8xEL1alzUFzYIGCfoQIPiTqeHbn5rUeEEH8LJnNKQDURgSlNB2lZU4SIii/N9Io3Z/AlsBO8KA4NZ9JcgRk7wzb71SEylcWVkwQW8F4jg81gLL8qCN0S4b9anBZ/KQVgcbYXRZjb2VWcXS36aHk5kHPesgtDikJbMnM1aCjv9blBnPRXEcHrht38bIctk6qUPeCs= ceph-3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:57:01.699 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.951+0000 7fd05fbcd640 1 Processor -- start 2024-09-10T18:57:01.699 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.951+0000 7fd05fbcd640 1 -- start start 2024-09-10T18:57:01.699 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.951+0000 7fd05fbcd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd058100670 0x7fd058100a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:01.699 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.951+0000 7fd05fbcd640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd058100fc0 con 0x7fd058100670 2024-09-10T18:57:01.699 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.951+0000 7fd05d969640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd058100670 0x7fd058100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:01.699 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.951+0000 7fd05d969640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd058100670 0x7fd058100a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49262/0 (socket says 172.21.15.119:49262) 2024-09-10T18:57:01.699 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.951+0000 7fd05d969640 1 -- 172.21.15.119:0/1956497494 learned_addr learned my addr 172.21.15.119:0/1956497494 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:01.699 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.952+0000 7fd05d969640 1 -- 172.21.15.119:0/1956497494 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd058101100 con 0x7fd058100670 2024-09-10T18:57:01.699 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.952+0000 7fd05d969640 1 --2- 172.21.15.119:0/1956497494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd058100670 0x7fd058100a80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd054009840 tx=0x7fd0540091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc3f443eca72ba1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:01.699 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.952+0000 7fd05c967640 1 -- 172.21.15.119:0/1956497494 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd05402add0 con 0x7fd058100670 2024-09-10T18:57:01.699 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.952+0000 7fd05c967640 1 -- 172.21.15.119:0/1956497494 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7fd054032890 con 0x7fd058100670 2024-09-10T18:57:01.700 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.952+0000 7fd05c967640 1 -- 172.21.15.119:0/1956497494 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0540317a0 con 0x7fd058100670 2024-09-10T18:57:01.700 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.953+0000 7fd05fbcd640 1 -- 172.21.15.119:0/1956497494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd058100670 msgr2=0x7fd058100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:01.700 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.953+0000 7fd05fbcd640 1 --2- 172.21.15.119:0/1956497494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd058100670 0x7fd058100a80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd054009840 tx=0x7fd0540091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:01.700 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.953+0000 7fd05fbcd640 1 -- 172.21.15.119:0/1956497494 shutdown_connections 2024-09-10T18:57:01.700 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.953+0000 7fd05fbcd640 1 --2- 172.21.15.119:0/1956497494 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd058100670 0x7fd058100a80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:01.700 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.953+0000 7fd05fbcd640 1 -- 172.21.15.119:0/1956497494 >> 172.21.15.119:0/1956497494 conn(0x7fd0580fbe10 msgr2=0x7fd0580fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:01.700 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.953+0000 7fd05fbcd640 1 -- 172.21.15.119:0/1956497494 shutdown_connections 2024-09-10T18:57:01.700 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.953+0000 7fd05fbcd640 1 -- 172.21.15.119:0/1956497494 wait complete. 2024-09-10T18:57:01.700 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.954+0000 7fd05fbcd640 1 Processor -- start 2024-09-10T18:57:01.700 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.954+0000 7fd05fbcd640 1 -- start start 2024-09-10T18:57:01.700 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.954+0000 7fd05fbcd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd058100670 0x7fd05810f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:01.700 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.954+0000 7fd05fbcd640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05810f7b0 con 0x7fd058100670 2024-09-10T18:57:01.700 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.954+0000 7fd05d969640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd058100670 0x7fd05810f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:01.701 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.954+0000 7fd05d969640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd058100670 0x7fd05810f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49268/0 (socket says 172.21.15.119:49268) 2024-09-10T18:57:01.701 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.954+0000 7fd05d969640 1 -- 172.21.15.119:0/456003335 learned_addr learned my addr 172.21.15.119:0/456003335 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:01.701 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.954+0000 7fd05d969640 1 -- 172.21.15.119:0/456003335 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd054009450 con 0x7fd058100670 2024-09-10T18:57:01.701 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.954+0000 7fd05d969640 1 --2- 172.21.15.119:0/456003335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd058100670 0x7fd05810f270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd0540313e0 tx=0x7fd0540091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:01.701 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.954+0000 7fd04e7fc640 1 -- 172.21.15.119:0/456003335 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd054031e70 con 0x7fd058100670 2024-09-10T18:57:01.701 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.954+0000 7fd04e7fc640 1 -- 172.21.15.119:0/456003335 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7fd05403c850 con 0x7fd058100670 2024-09-10T18:57:01.701 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.954+0000 7fd05fbcd640 1 -- 172.21.15.119:0/456003335 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05810f9b0 con 0x7fd058100670 2024-09-10T18:57:01.701 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.954+0000 7fd04e7fc640 1 -- 172.21.15.119:0/456003335 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd05403a540 con 0x7fd058100670 2024-09-10T18:57:01.701 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.955+0000 7fd05fbcd640 1 -- 172.21.15.119:0/456003335 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05810fe20 con 0x7fd058100670 2024-09-10T18:57:01.701 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.955+0000 7fd04e7fc640 1 -- 172.21.15.119:0/456003335 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 45457+0+0 (secure 0 0 0) 0x7fd05403cc10 con 0x7fd058100670 2024-09-10T18:57:01.701 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.956+0000 7fd04e7fc640 1 --2- 172.21.15.119:0/456003335 >> v2:172.21.15.119:6800/1136125279 conn(0x7fd044039470 0x7fd04403b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:01.701 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.956+0000 7fd05fbcd640 1 -- 172.21.15.119:0/456003335 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd05804b720 con 0x7fd058100670 2024-09-10T18:57:01.701 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.956+0000 7fd04e7fc640 1 -- 172.21.15.119:0/456003335 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fd05406c5f0 con 0x7fd058100670 2024-09-10T18:57:01.702 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.956+0000 7fd05d168640 1 --2- 172.21.15.119:0/456003335 >> v2:172.21.15.119:6800/1136125279 conn(0x7fd044039470 0x7fd04403b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:01.702 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.956+0000 7fd05d168640 1 --2- 172.21.15.119:0/456003335 >> v2:172.21.15.119:6800/1136125279 conn(0x7fd044039470 0x7fd04403b920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd048006fd0 tx=0x7fd048006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:01.702 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:00.960+0000 7fd04e7fc640 1 -- 172.21.15.119:0/456003335 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fd05403aad0 con 0x7fd058100670 2024-09-10T18:57:01.702 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:01.115+0000 7fd05fbcd640 1 -- 172.21.15.119:0/456003335 --> v2:172.21.15.119:6800/1136125279 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fd05810fe20 con 0x7fd044039470 2024-09-10T18:57:01.702 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:01.116+0000 7fd04e7fc640 1 -- 172.21.15.119:0/456003335 <== mgr.14118 v2:172.21.15.119:6800/1136125279 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fd05810fe20 con 0x7fd044039470 2024-09-10T18:57:01.702 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:01.118+0000 7fd05fbcd640 1 -- 172.21.15.119:0/456003335 >> v2:172.21.15.119:6800/1136125279 conn(0x7fd044039470 msgr2=0x7fd04403b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:01.704 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:01.118+0000 7fd05fbcd640 1 --2- 172.21.15.119:0/456003335 >> v2:172.21.15.119:6800/1136125279 conn(0x7fd044039470 0x7fd04403b920 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd048006fd0 tx=0x7fd048006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:01.704 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:01.118+0000 7fd05fbcd640 1 -- 172.21.15.119:0/456003335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd058100670 msgr2=0x7fd05810f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:01.704 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:01.118+0000 7fd05fbcd640 1 --2- 172.21.15.119:0/456003335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd058100670 0x7fd05810f270 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd0540313e0 tx=0x7fd0540091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:01.704 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:01.118+0000 7fd05fbcd640 1 -- 172.21.15.119:0/456003335 shutdown_connections 2024-09-10T18:57:01.704 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:01.118+0000 7fd05fbcd640 1 --2- 172.21.15.119:0/456003335 >> v2:172.21.15.119:6800/1136125279 conn(0x7fd044039470 0x7fd04403b920 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:01.704 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:01.118+0000 7fd05fbcd640 1 --2- 172.21.15.119:0/456003335 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd058100670 0x7fd05810f270 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:01.704 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:01.118+0000 7fd05fbcd640 1 -- 172.21.15.119:0/456003335 >> 172.21.15.119:0/456003335 conn(0x7fd0580fbe10 msgr2=0x7fd0580fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:01.704 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:01.118+0000 7fd05fbcd640 1 -- 172.21.15.119:0/456003335 shutdown_connections 2024-09-10T18:57:01.705 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:01.118+0000 7fd05fbcd640 1 -- 172.21.15.119:0/456003335 wait complete. 2024-09-10T18:57:01.705 INFO:teuthology.orchestra.run.smithi119.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-10T18:57:01.705 INFO:teuthology.orchestra.run.smithi119.stdout:Adding key to root@localhost authorized_keys... 2024-09-10T18:57:01.707 INFO:teuthology.orchestra.run.smithi119.stdout:Adding host smithi119... 2024-09-10T18:57:02.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:01 smithi119 ceph-mon[29577]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:57:03.103 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:02 smithi119 ceph-mon[29577]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi119", "addr": "172.21.15.119", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:57:04.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:03 smithi119 ceph-mon[29577]: Deploying cephadm binary to smithi119 2024-09-10T18:57:05.548 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout Added host 'smithi119' with addr '172.21.15.119' 2024-09-10T18:57:05.548 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.144+0000 7fe6eed9d640 1 Processor -- start 2024-09-10T18:57:05.549 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.144+0000 7fe6eed9d640 1 -- start start 2024-09-10T18:57:05.549 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.144+0000 7fe6eed9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6e81006c0 0x7fe6e8100ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:05.549 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.144+0000 7fe6eed9d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e8101010 con 0x7fe6e81006c0 2024-09-10T18:57:05.549 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.144+0000 7fe6ecb39640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6e81006c0 0x7fe6e8100ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:05.549 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.145+0000 7fe6ecb39640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6e81006c0 0x7fe6e8100ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49274/0 (socket says 172.21.15.119:49274) 2024-09-10T18:57:05.549 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.145+0000 7fe6ecb39640 1 -- 172.21.15.119:0/2367909866 learned_addr learned my addr 172.21.15.119:0/2367909866 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:05.549 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.145+0000 7fe6ecb39640 1 -- 172.21.15.119:0/2367909866 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e8101150 con 0x7fe6e81006c0 2024-09-10T18:57:05.549 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.145+0000 7fe6ecb39640 1 --2- 172.21.15.119:0/2367909866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6e81006c0 0x7fe6e8100ad0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8009e50 tx=0x7fe6d8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=770166a99df22bcf server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:05.549 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.146+0000 7fe6e77fe640 1 -- 172.21.15.119:0/2367909866 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6d8031400 con 0x7fe6e81006c0 2024-09-10T18:57:05.549 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.146+0000 7fe6e77fe640 1 -- 172.21.15.119:0/2367909866 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7fe6d802d050 con 0x7fe6e81006c0 2024-09-10T18:57:05.549 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.146+0000 7fe6e77fe640 1 -- 172.21.15.119:0/2367909866 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6d80076e0 con 0x7fe6e81006c0 2024-09-10T18:57:05.550 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.146+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2367909866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6e81006c0 msgr2=0x7fe6e8100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:05.550 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.146+0000 7fe6eed9d640 1 --2- 172.21.15.119:0/2367909866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6e81006c0 0x7fe6e8100ad0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8009e50 tx=0x7fe6d8009840 comp rx=0 tx=0).stop 2024-09-10T18:57:05.550 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.146+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2367909866 shutdown_connections 2024-09-10T18:57:05.550 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.146+0000 7fe6eed9d640 1 --2- 172.21.15.119:0/2367909866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6e81006c0 0x7fe6e8100ad0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:05.550 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.146+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2367909866 >> 172.21.15.119:0/2367909866 conn(0x7fe6e80fbea0 msgr2=0x7fe6e80fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:05.550 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.146+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2367909866 shutdown_connections 2024-09-10T18:57:05.550 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.146+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2367909866 wait complete. 2024-09-10T18:57:05.550 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.147+0000 7fe6eed9d640 1 Processor -- start 2024-09-10T18:57:05.551 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.147+0000 7fe6eed9d640 1 -- start start 2024-09-10T18:57:05.551 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.147+0000 7fe6eed9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6e81006c0 0x7fe6e80774b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:05.551 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.147+0000 7fe6eed9d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e80779f0 con 0x7fe6e81006c0 2024-09-10T18:57:05.551 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.147+0000 7fe6ecb39640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6e81006c0 0x7fe6e80774b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:05.551 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.147+0000 7fe6ecb39640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6e81006c0 0x7fe6e80774b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49282/0 (socket says 172.21.15.119:49282) 2024-09-10T18:57:05.551 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.147+0000 7fe6ecb39640 1 -- 172.21.15.119:0/2063542795 learned_addr learned my addr 172.21.15.119:0/2063542795 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:05.551 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.147+0000 7fe6ecb39640 1 -- 172.21.15.119:0/2063542795 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d8009ae0 con 0x7fe6e81006c0 2024-09-10T18:57:05.551 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.148+0000 7fe6ecb39640 1 --2- 172.21.15.119:0/2063542795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6e81006c0 0x7fe6e80774b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe6d802ac70 tx=0x7fe6d8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:05.551 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.148+0000 7fe6e57fa640 1 -- 172.21.15.119:0/2063542795 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6d80078c0 con 0x7fe6e81006c0 2024-09-10T18:57:05.551 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.148+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2063542795 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e8075a80 con 0x7fe6e81006c0 2024-09-10T18:57:05.551 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.148+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2063542795 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e8075ef0 con 0x7fe6e81006c0 2024-09-10T18:57:05.552 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.148+0000 7fe6e57fa640 1 -- 172.21.15.119:0/2063542795 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7fe6d8007e70 con 0x7fe6e81006c0 2024-09-10T18:57:05.552 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.148+0000 7fe6e57fa640 1 -- 172.21.15.119:0/2063542795 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe6d8045c10 con 0x7fe6e81006c0 2024-09-10T18:57:05.552 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.149+0000 7fe6e57fa640 1 -- 172.21.15.119:0/2063542795 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 45457+0+0 (secure 0 0 0) 0x7fe6d8038040 con 0x7fe6e81006c0 2024-09-10T18:57:05.552 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.149+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2063542795 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c8005040 con 0x7fe6e81006c0 2024-09-10T18:57:05.552 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.149+0000 7fe6e57fa640 1 --2- 172.21.15.119:0/2063542795 >> v2:172.21.15.119:6800/1136125279 conn(0x7fe6d4039430 0x7fe6d403b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:05.552 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.149+0000 7fe6e57fa640 1 -- 172.21.15.119:0/2063542795 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fe6d8040030 con 0x7fe6e81006c0 2024-09-10T18:57:05.552 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.149+0000 7fe6e7fff640 1 --2- 172.21.15.119:0/2063542795 >> v2:172.21.15.119:6800/1136125279 conn(0x7fe6d4039430 0x7fe6d403b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:05.552 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.149+0000 7fe6e7fff640 1 --2- 172.21.15.119:0/2063542795 >> v2:172.21.15.119:6800/1136125279 conn(0x7fe6d4039430 0x7fe6d403b8e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0006fd0 tx=0x7fe6d0006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:05.552 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.153+0000 7fe6e57fa640 1 -- 172.21.15.119:0/2063542795 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fe6d804e3f0 con 0x7fe6e81006c0 2024-09-10T18:57:05.552 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:02.315+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2063542795 --> v2:172.21.15.119:6800/1136125279 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi119", "addr": "172.21.15.119", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6c8000bf0 con 0x7fe6d4039430 2024-09-10T18:57:05.552 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:04.959+0000 7fe6e57fa640 1 -- 172.21.15.119:0/2063542795 <== mgr.14118 v2:172.21.15.119:6800/1136125279 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fe6c8000bf0 con 0x7fe6d4039430 2024-09-10T18:57:05.552 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:04.963+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2063542795 >> v2:172.21.15.119:6800/1136125279 conn(0x7fe6d4039430 msgr2=0x7fe6d403b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:05.552 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:04.963+0000 7fe6eed9d640 1 --2- 172.21.15.119:0/2063542795 >> v2:172.21.15.119:6800/1136125279 conn(0x7fe6d4039430 0x7fe6d403b8e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe6d0006fd0 tx=0x7fe6d0006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:05.553 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:04.963+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2063542795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6e81006c0 msgr2=0x7fe6e80774b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:05.553 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:04.963+0000 7fe6eed9d640 1 --2- 172.21.15.119:0/2063542795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6e81006c0 0x7fe6e80774b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe6d802ac70 tx=0x7fe6d8009840 comp rx=0 tx=0).stop 2024-09-10T18:57:05.553 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:04.963+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2063542795 shutdown_connections 2024-09-10T18:57:05.553 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:04.963+0000 7fe6eed9d640 1 --2- 172.21.15.119:0/2063542795 >> v2:172.21.15.119:6800/1136125279 conn(0x7fe6d4039430 0x7fe6d403b8e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:05.553 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:04.963+0000 7fe6eed9d640 1 --2- 172.21.15.119:0/2063542795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6e81006c0 0x7fe6e80774b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:05.553 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:04.963+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2063542795 >> 172.21.15.119:0/2063542795 conn(0x7fe6e80fbea0 msgr2=0x7fe6e80fc9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:05.553 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:04.963+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2063542795 shutdown_connections 2024-09-10T18:57:05.553 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:04.963+0000 7fe6eed9d640 1 -- 172.21.15.119:0/2063542795 wait complete. 2024-09-10T18:57:05.553 INFO:teuthology.orchestra.run.smithi119.stdout:Deploying unmanaged mon service... 2024-09-10T18:57:06.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:05 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' 2024-09-10T18:57:06.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:05 smithi119 ceph-mon[29577]: Added host smithi119 2024-09-10T18:57:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:05 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:57:07.461 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:07 smithi119 ceph-mon[29577]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:57:07.461 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:07 smithi119 ceph-mon[29577]: Saving service mon spec with placement count:5 2024-09-10T18:57:07.461 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:07 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' 2024-09-10T18:57:07.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-10T18:57:07.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.005+0000 7f01537f7640 1 Processor -- start 2024-09-10T18:57:07.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.005+0000 7f01537f7640 1 -- start start 2024-09-10T18:57:07.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.005+0000 7f01537f7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f014c101c20 0x7f014c102030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:07.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.005+0000 7f01537f7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014c102570 con 0x7f014c101c20 2024-09-10T18:57:07.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.005+0000 7f0151593640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f014c101c20 0x7f014c102030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:07.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.005+0000 7f0151593640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f014c101c20 0x7f014c102030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36650/0 (socket says 172.21.15.119:36650) 2024-09-10T18:57:07.838 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.005+0000 7f0151593640 1 -- 172.21.15.119:0/3913197659 learned_addr learned my addr 172.21.15.119:0/3913197659 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.006+0000 7f0151593640 1 -- 172.21.15.119:0/3913197659 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c1026b0 con 0x7f014c101c20 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.006+0000 7f0151593640 1 --2- 172.21.15.119:0/3913197659 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f014c101c20 0x7f014c102030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0140009840 tx=0x7f01400091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e979536312c1853d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.006+0000 7f013ffff640 1 -- 172.21.15.119:0/3913197659 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f014002add0 con 0x7f014c101c20 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.006+0000 7f013ffff640 1 -- 172.21.15.119:0/3913197659 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f0140032890 con 0x7f014c101c20 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.006+0000 7f013ffff640 1 -- 172.21.15.119:0/3913197659 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01400317a0 con 0x7f014c101c20 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.007+0000 7f01537f7640 1 -- 172.21.15.119:0/3913197659 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f014c101c20 msgr2=0x7f014c102030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.007+0000 7f01537f7640 1 --2- 172.21.15.119:0/3913197659 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f014c101c20 0x7f014c102030 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0140009840 tx=0x7f01400091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.007+0000 7f01537f7640 1 -- 172.21.15.119:0/3913197659 shutdown_connections 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.007+0000 7f01537f7640 1 --2- 172.21.15.119:0/3913197659 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f014c101c20 0x7f014c102030 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.007+0000 7f01537f7640 1 -- 172.21.15.119:0/3913197659 >> 172.21.15.119:0/3913197659 conn(0x7f014c0fa340 msgr2=0x7f014c0fc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.007+0000 7f01537f7640 1 -- 172.21.15.119:0/3913197659 shutdown_connections 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.007+0000 7f01537f7640 1 -- 172.21.15.119:0/3913197659 wait complete. 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.008+0000 7f01537f7640 1 Processor -- start 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.008+0000 7f01537f7640 1 -- start start 2024-09-10T18:57:07.839 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.008+0000 7f01537f7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f014c101c20 0x7f014c10f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:07.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.008+0000 7f01537f7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f014c10f800 con 0x7f014c101c20 2024-09-10T18:57:07.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.008+0000 7f0151593640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f014c101c20 0x7f014c10f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:07.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.008+0000 7f0151593640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f014c101c20 0x7f014c10f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36664/0 (socket says 172.21.15.119:36664) 2024-09-10T18:57:07.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.008+0000 7f0151593640 1 -- 172.21.15.119:0/1448969078 learned_addr learned my addr 172.21.15.119:0/1448969078 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:07.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.008+0000 7f0151593640 1 -- 172.21.15.119:0/1448969078 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0140009450 con 0x7f014c101c20 2024-09-10T18:57:07.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.008+0000 7f0151593640 1 --2- 172.21.15.119:0/1448969078 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f014c101c20 0x7f014c10f2c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f01400313e0 tx=0x7f01400091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:07.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.008+0000 7f013dffb640 1 -- 172.21.15.119:0/1448969078 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0140031e70 con 0x7f014c101c20 2024-09-10T18:57:07.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.008+0000 7f013dffb640 1 -- 172.21.15.119:0/1448969078 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 988+0+0 (secure 0 0 0) 0x7f014003c850 con 0x7f014c101c20 2024-09-10T18:57:07.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.008+0000 7f01537f7640 1 -- 172.21.15.119:0/1448969078 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014c10fa00 con 0x7f014c101c20 2024-09-10T18:57:07.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.008+0000 7f013dffb640 1 -- 172.21.15.119:0/1448969078 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f014003a540 con 0x7f014c101c20 2024-09-10T18:57:07.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.009+0000 7f01537f7640 1 -- 172.21.15.119:0/1448969078 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014c10fe70 con 0x7f014c101c20 2024-09-10T18:57:07.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.009+0000 7f013dffb640 1 -- 172.21.15.119:0/1448969078 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 45457+0+0 (secure 0 0 0) 0x7f014003a7c0 con 0x7f014c101c20 2024-09-10T18:57:07.840 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.009+0000 7f01537f7640 1 -- 172.21.15.119:0/1448969078 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014c04b720 con 0x7f014c101c20 2024-09-10T18:57:07.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.010+0000 7f013dffb640 1 --2- 172.21.15.119:0/1448969078 >> v2:172.21.15.119:6800/1136125279 conn(0x7f0138039430 0x7f013803b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:07.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.010+0000 7f013dffb640 1 -- 172.21.15.119:0/1448969078 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f014006c5d0 con 0x7f014c101c20 2024-09-10T18:57:07.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.010+0000 7f0150d92640 1 --2- 172.21.15.119:0/1448969078 >> v2:172.21.15.119:6800/1136125279 conn(0x7f0138039430 0x7f013803b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:07.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.010+0000 7f0150d92640 1 --2- 172.21.15.119:0/1448969078 >> v2:172.21.15.119:6800/1136125279 conn(0x7f0138039430 0x7f013803b8e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0148006fd0 tx=0x7f0148006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:07.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.014+0000 7f013dffb640 1 -- 172.21.15.119:0/1448969078 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f0140039960 con 0x7f014c101c20 2024-09-10T18:57:07.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.181+0000 7f01537f7640 1 -- 172.21.15.119:0/1448969078 --> v2:172.21.15.119:6800/1136125279 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f014c10fe70 con 0x7f0138039430 2024-09-10T18:57:07.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.188+0000 7f013dffb640 1 -- 172.21.15.119:0/1448969078 <== mgr.14118 v2:172.21.15.119:6800/1136125279 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f014c10fe70 con 0x7f0138039430 2024-09-10T18:57:07.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.190+0000 7f01537f7640 1 -- 172.21.15.119:0/1448969078 >> v2:172.21.15.119:6800/1136125279 conn(0x7f0138039430 msgr2=0x7f013803b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.841 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.190+0000 7f01537f7640 1 --2- 172.21.15.119:0/1448969078 >> v2:172.21.15.119:6800/1136125279 conn(0x7f0138039430 0x7f013803b8e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0148006fd0 tx=0x7f0148006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:07.842 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.190+0000 7f01537f7640 1 -- 172.21.15.119:0/1448969078 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f014c101c20 msgr2=0x7f014c10f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:07.842 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.190+0000 7f01537f7640 1 --2- 172.21.15.119:0/1448969078 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f014c101c20 0x7f014c10f2c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f01400313e0 tx=0x7f01400091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:07.842 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.190+0000 7f01537f7640 1 -- 172.21.15.119:0/1448969078 shutdown_connections 2024-09-10T18:57:07.842 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.190+0000 7f01537f7640 1 --2- 172.21.15.119:0/1448969078 >> v2:172.21.15.119:6800/1136125279 conn(0x7f0138039430 0x7f013803b8e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:07.842 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.190+0000 7f01537f7640 1 --2- 172.21.15.119:0/1448969078 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f014c101c20 0x7f014c10f2c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:07.842 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.190+0000 7f01537f7640 1 -- 172.21.15.119:0/1448969078 >> 172.21.15.119:0/1448969078 conn(0x7f014c0fa340 msgr2=0x7f014c0fae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:07.842 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.190+0000 7f01537f7640 1 -- 172.21.15.119:0/1448969078 shutdown_connections 2024-09-10T18:57:07.842 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:06.190+0000 7f01537f7640 1 -- 172.21.15.119:0/1448969078 wait complete. 2024-09-10T18:57:07.842 INFO:teuthology.orchestra.run.smithi119.stdout:Deploying unmanaged mgr service... 2024-09-10T18:57:08.745 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:08 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' 2024-09-10T18:57:08.745 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:08 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' 2024-09-10T18:57:09.724 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:09 smithi119 ceph-mon[29577]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:57:09.725 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:09 smithi119 ceph-mon[29577]: Saving service mgr spec with placement count:2 2024-09-10T18:57:09.725 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:09 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' 2024-09-10T18:57:09.725 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:09 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' 2024-09-10T18:57:09.731 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-10T18:57:09.731 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.292+0000 7fcd9375f640 1 Processor -- start 2024-09-10T18:57:09.731 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.292+0000 7fcd9375f640 1 -- start start 2024-09-10T18:57:09.731 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.293+0000 7fcd9375f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd8c074c40 0x7fcd8c103190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:09.731 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.293+0000 7fcd9375f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c1036d0 con 0x7fcd8c074c40 2024-09-10T18:57:09.731 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.293+0000 7fcd914fb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd8c074c40 0x7fcd8c103190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:09.732 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.293+0000 7fcd914fb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd8c074c40 0x7fcd8c103190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36676/0 (socket says 172.21.15.119:36676) 2024-09-10T18:57:09.732 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.293+0000 7fcd914fb640 1 -- 172.21.15.119:0/3495940127 learned_addr learned my addr 172.21.15.119:0/3495940127 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:09.732 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.293+0000 7fcd914fb640 1 -- 172.21.15.119:0/3495940127 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8c103810 con 0x7fcd8c074c40 2024-09-10T18:57:09.732 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.294+0000 7fcd914fb640 1 --2- 172.21.15.119:0/3495940127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd8c074c40 0x7fcd8c103190 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcd80009840 tx=0x7fcd800091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cda9efdf68eb124 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:09.732 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.294+0000 7fcd7ffff640 1 -- 172.21.15.119:0/3495940127 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd8002add0 con 0x7fcd8c074c40 2024-09-10T18:57:09.732 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.294+0000 7fcd7ffff640 1 -- 172.21.15.119:0/3495940127 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fcd800328b0 con 0x7fcd8c074c40 2024-09-10T18:57:09.732 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.294+0000 7fcd7ffff640 1 -- 172.21.15.119:0/3495940127 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd80031690 con 0x7fcd8c074c40 2024-09-10T18:57:09.732 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.294+0000 7fcd9375f640 1 -- 172.21.15.119:0/3495940127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd8c074c40 msgr2=0x7fcd8c103190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:09.732 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.294+0000 7fcd9375f640 1 --2- 172.21.15.119:0/3495940127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd8c074c40 0x7fcd8c103190 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fcd80009840 tx=0x7fcd800091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:09.732 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.294+0000 7fcd9375f640 1 -- 172.21.15.119:0/3495940127 shutdown_connections 2024-09-10T18:57:09.732 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.294+0000 7fcd9375f640 1 --2- 172.21.15.119:0/3495940127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd8c074c40 0x7fcd8c103190 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:09.733 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.294+0000 7fcd9375f640 1 -- 172.21.15.119:0/3495940127 >> 172.21.15.119:0/3495940127 conn(0x7fcd8c0fabc0 msgr2=0x7fcd8c0fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:09.733 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.295+0000 7fcd9375f640 1 -- 172.21.15.119:0/3495940127 shutdown_connections 2024-09-10T18:57:09.733 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.295+0000 7fcd9375f640 1 -- 172.21.15.119:0/3495940127 wait complete. 2024-09-10T18:57:09.733 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.295+0000 7fcd9375f640 1 Processor -- start 2024-09-10T18:57:09.733 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.295+0000 7fcd9375f640 1 -- start start 2024-09-10T18:57:09.733 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.295+0000 7fcd9375f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd8c074c40 0x7fcd8c10f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:09.733 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.295+0000 7fcd9375f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c10f850 con 0x7fcd8c074c40 2024-09-10T18:57:09.733 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.295+0000 7fcd914fb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd8c074c40 0x7fcd8c10f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:09.733 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.295+0000 7fcd914fb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd8c074c40 0x7fcd8c10f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36684/0 (socket says 172.21.15.119:36684) 2024-09-10T18:57:09.733 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.295+0000 7fcd914fb640 1 -- 172.21.15.119:0/2177460580 learned_addr learned my addr 172.21.15.119:0/2177460580 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:09.733 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.295+0000 7fcd914fb640 1 -- 172.21.15.119:0/2177460580 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd80009450 con 0x7fcd8c074c40 2024-09-10T18:57:09.733 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.296+0000 7fcd914fb640 1 --2- 172.21.15.119:0/2177460580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd8c074c40 0x7fcd8c10f310 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcd80006f00 tx=0x7fcd80008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:09.733 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.296+0000 7fcd7dffb640 1 -- 172.21.15.119:0/2177460580 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd80031c60 con 0x7fcd8c074c40 2024-09-10T18:57:09.734 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.296+0000 7fcd7dffb640 1 -- 172.21.15.119:0/2177460580 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fcd80031dc0 con 0x7fcd8c074c40 2024-09-10T18:57:09.734 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.296+0000 7fcd9375f640 1 -- 172.21.15.119:0/2177460580 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd8c10fa50 con 0x7fcd8c074c40 2024-09-10T18:57:09.734 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.296+0000 7fcd7dffb640 1 -- 172.21.15.119:0/2177460580 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcd8003a540 con 0x7fcd8c074c40 2024-09-10T18:57:09.734 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.296+0000 7fcd9375f640 1 -- 172.21.15.119:0/2177460580 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8c10fec0 con 0x7fcd8c074c40 2024-09-10T18:57:09.734 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.297+0000 7fcd7dffb640 1 -- 172.21.15.119:0/2177460580 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 45457+0+0 (secure 0 0 0) 0x7fcd80039070 con 0x7fcd8c074c40 2024-09-10T18:57:09.734 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.297+0000 7fcd7dffb640 1 --2- 172.21.15.119:0/2177460580 >> v2:172.21.15.119:6800/1136125279 conn(0x7fcd78039470 0x7fcd7803b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:09.734 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.297+0000 7fcd9375f640 1 -- 172.21.15.119:0/2177460580 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd8c04b720 con 0x7fcd8c074c40 2024-09-10T18:57:09.734 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.297+0000 7fcd7dffb640 1 -- 172.21.15.119:0/2177460580 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fcd8006c7f0 con 0x7fcd8c074c40 2024-09-10T18:57:09.734 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.297+0000 7fcd90cfa640 1 --2- 172.21.15.119:0/2177460580 >> v2:172.21.15.119:6800/1136125279 conn(0x7fcd78039470 0x7fcd7803b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:09.734 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.298+0000 7fcd90cfa640 1 --2- 172.21.15.119:0/2177460580 >> v2:172.21.15.119:6800/1136125279 conn(0x7fcd78039470 0x7fcd7803b920 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcd88006fd0 tx=0x7fcd88006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:09.734 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.302+0000 7fcd7dffb640 1 -- 172.21.15.119:0/2177460580 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fcd80037020 con 0x7fcd8c074c40 2024-09-10T18:57:09.734 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.484+0000 7fcd9375f640 1 -- 172.21.15.119:0/2177460580 --> v2:172.21.15.119:6800/1136125279 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fcd8c0fbf00 con 0x7fcd78039470 2024-09-10T18:57:09.734 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.495+0000 7fcd7dffb640 1 -- 172.21.15.119:0/2177460580 <== mgr.14118 v2:172.21.15.119:6800/1136125279 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fcd8c0fbf00 con 0x7fcd78039470 2024-09-10T18:57:09.735 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.497+0000 7fcd9375f640 1 -- 172.21.15.119:0/2177460580 >> v2:172.21.15.119:6800/1136125279 conn(0x7fcd78039470 msgr2=0x7fcd7803b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:09.735 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.497+0000 7fcd9375f640 1 --2- 172.21.15.119:0/2177460580 >> v2:172.21.15.119:6800/1136125279 conn(0x7fcd78039470 0x7fcd7803b920 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcd88006fd0 tx=0x7fcd88006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:09.735 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.497+0000 7fcd9375f640 1 -- 172.21.15.119:0/2177460580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd8c074c40 msgr2=0x7fcd8c10f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:09.735 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.497+0000 7fcd9375f640 1 --2- 172.21.15.119:0/2177460580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd8c074c40 0x7fcd8c10f310 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fcd80006f00 tx=0x7fcd80008040 comp rx=0 tx=0).stop 2024-09-10T18:57:09.735 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.497+0000 7fcd9375f640 1 -- 172.21.15.119:0/2177460580 shutdown_connections 2024-09-10T18:57:09.735 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.497+0000 7fcd9375f640 1 --2- 172.21.15.119:0/2177460580 >> v2:172.21.15.119:6800/1136125279 conn(0x7fcd78039470 0x7fcd7803b920 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:09.735 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.497+0000 7fcd9375f640 1 --2- 172.21.15.119:0/2177460580 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd8c074c40 0x7fcd8c10f310 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:09.735 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.497+0000 7fcd9375f640 1 -- 172.21.15.119:0/2177460580 >> 172.21.15.119:0/2177460580 conn(0x7fcd8c0fabc0 msgr2=0x7fcd8c0fb7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:09.735 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.498+0000 7fcd9375f640 1 -- 172.21.15.119:0/2177460580 shutdown_connections 2024-09-10T18:57:09.735 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:08.498+0000 7fcd9375f640 1 -- 172.21.15.119:0/2177460580 wait complete. 2024-09-10T18:57:11.002 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.241+0000 7f4c447b2640 1 Processor -- start 2024-09-10T18:57:11.003 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.241+0000 7f4c447b2640 1 -- start start 2024-09-10T18:57:11.003 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.241+0000 7f4c447b2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c3c100690 0x7f4c3c100aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:11.003 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.241+0000 7f4c447b2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c100fe0 con 0x7f4c3c100690 2024-09-10T18:57:11.003 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.241+0000 7f4c4254e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c3c100690 0x7f4c3c100aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:11.003 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.241+0000 7f4c4254e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c3c100690 0x7f4c3c100aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36686/0 (socket says 172.21.15.119:36686) 2024-09-10T18:57:11.004 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.241+0000 7f4c4254e640 1 -- 172.21.15.119:0/3020891486 learned_addr learned my addr 172.21.15.119:0/3020891486 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:11.004 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.242+0000 7f4c4254e640 1 -- 172.21.15.119:0/3020891486 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c101120 con 0x7f4c3c100690 2024-09-10T18:57:11.004 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.242+0000 7f4c4254e640 1 --2- 172.21.15.119:0/3020891486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c3c100690 0x7f4c3c100aa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009840 tx=0x7f4c380091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f8f555a78370a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:11.004 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.242+0000 7f4c4154c640 1 -- 172.21.15.119:0/3020891486 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4c3802ae60 con 0x7f4c3c100690 2024-09-10T18:57:11.004 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.243+0000 7f4c4154c640 1 -- 172.21.15.119:0/3020891486 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f4c380328b0 con 0x7f4c3c100690 2024-09-10T18:57:11.004 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.243+0000 7f4c4154c640 1 -- 172.21.15.119:0/3020891486 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4c38031740 con 0x7f4c3c100690 2024-09-10T18:57:11.005 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.243+0000 7f4c447b2640 1 -- 172.21.15.119:0/3020891486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c3c100690 msgr2=0x7f4c3c100aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:11.005 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.243+0000 7f4c447b2640 1 --2- 172.21.15.119:0/3020891486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c3c100690 0x7f4c3c100aa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4c38009840 tx=0x7f4c380091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:11.005 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.243+0000 7f4c447b2640 1 -- 172.21.15.119:0/3020891486 shutdown_connections 2024-09-10T18:57:11.005 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.243+0000 7f4c447b2640 1 --2- 172.21.15.119:0/3020891486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c3c100690 0x7f4c3c100aa0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:11.005 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.243+0000 7f4c447b2640 1 -- 172.21.15.119:0/3020891486 >> 172.21.15.119:0/3020891486 conn(0x7f4c3c0fbe10 msgr2=0x7f4c3c0fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:11.005 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.243+0000 7f4c447b2640 1 -- 172.21.15.119:0/3020891486 shutdown_connections 2024-09-10T18:57:11.006 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.243+0000 7f4c447b2640 1 -- 172.21.15.119:0/3020891486 wait complete. 2024-09-10T18:57:11.006 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.244+0000 7f4c447b2640 1 Processor -- start 2024-09-10T18:57:11.006 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.244+0000 7f4c447b2640 1 -- start start 2024-09-10T18:57:11.006 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.244+0000 7f4c447b2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c3c100690 0x7f4c3c10f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:11.006 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.244+0000 7f4c447b2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c10f6b0 con 0x7f4c3c100690 2024-09-10T18:57:11.006 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.244+0000 7f4c4254e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c3c100690 0x7f4c3c10f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:11.007 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.244+0000 7f4c4254e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c3c100690 0x7f4c3c10f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36688/0 (socket says 172.21.15.119:36688) 2024-09-10T18:57:11.007 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.244+0000 7f4c4254e640 1 -- 172.21.15.119:0/560885962 learned_addr learned my addr 172.21.15.119:0/560885962 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:11.007 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.244+0000 7f4c4254e640 1 -- 172.21.15.119:0/560885962 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c38009450 con 0x7f4c3c100690 2024-09-10T18:57:11.007 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.245+0000 7f4c4254e640 1 --2- 172.21.15.119:0/560885962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c3c100690 0x7f4c3c10f170 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4c380326a0 tx=0x7f4c380091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:11.007 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.245+0000 7f4c32ffd640 1 -- 172.21.15.119:0/560885962 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4c38031bc0 con 0x7f4c3c100690 2024-09-10T18:57:11.007 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.245+0000 7f4c32ffd640 1 -- 172.21.15.119:0/560885962 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f4c38031d20 con 0x7f4c3c100690 2024-09-10T18:57:11.008 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.245+0000 7f4c32ffd640 1 -- 172.21.15.119:0/560885962 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4c3803be50 con 0x7f4c3c100690 2024-09-10T18:57:11.008 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.245+0000 7f4c447b2640 1 -- 172.21.15.119:0/560885962 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3c10f8b0 con 0x7f4c3c100690 2024-09-10T18:57:11.008 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.245+0000 7f4c447b2640 1 -- 172.21.15.119:0/560885962 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3c10fd20 con 0x7f4c3c100690 2024-09-10T18:57:11.008 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.246+0000 7f4c447b2640 1 -- 172.21.15.119:0/560885962 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c3c04b720 con 0x7f4c3c100690 2024-09-10T18:57:11.008 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.246+0000 7f4c32ffd640 1 -- 172.21.15.119:0/560885962 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 45457+0+0 (secure 0 0 0) 0x7f4c38039040 con 0x7f4c3c100690 2024-09-10T18:57:11.008 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.246+0000 7f4c32ffd640 1 --2- 172.21.15.119:0/560885962 >> v2:172.21.15.119:6800/1136125279 conn(0x7f4c280393e0 0x7f4c2803b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:11.009 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.246+0000 7f4c32ffd640 1 -- 172.21.15.119:0/560885962 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f4c3806c520 con 0x7f4c3c100690 2024-09-10T18:57:11.009 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.246+0000 7f4c41d4d640 1 --2- 172.21.15.119:0/560885962 >> v2:172.21.15.119:6800/1136125279 conn(0x7f4c280393e0 0x7f4c2803b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:11.009 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.247+0000 7f4c41d4d640 1 --2- 172.21.15.119:0/560885962 >> v2:172.21.15.119:6800/1136125279 conn(0x7f4c280393e0 0x7f4c2803b890 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c006fd0 tx=0x7f4c2c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:11.009 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.251+0000 7f4c32ffd640 1 -- 172.21.15.119:0/560885962 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f4c38030570 con 0x7f4c3c100690 2024-09-10T18:57:11.010 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.415+0000 7f4c447b2640 1 -- 172.21.15.119:0/560885962 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f4c3c102370 con 0x7f4c3c100690 2024-09-10T18:57:11.010 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.421+0000 7f4c32ffd640 1 -- 172.21.15.119:0/560885962 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f4c3803f070 con 0x7f4c3c100690 2024-09-10T18:57:11.010 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.424+0000 7f4c447b2640 1 -- 172.21.15.119:0/560885962 >> v2:172.21.15.119:6800/1136125279 conn(0x7f4c280393e0 msgr2=0x7f4c2803b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:11.010 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.424+0000 7f4c447b2640 1 --2- 172.21.15.119:0/560885962 >> v2:172.21.15.119:6800/1136125279 conn(0x7f4c280393e0 0x7f4c2803b890 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c006fd0 tx=0x7f4c2c006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:11.011 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.424+0000 7f4c447b2640 1 -- 172.21.15.119:0/560885962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c3c100690 msgr2=0x7f4c3c10f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:11.011 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.424+0000 7f4c447b2640 1 --2- 172.21.15.119:0/560885962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c3c100690 0x7f4c3c10f170 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4c380326a0 tx=0x7f4c380091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:11.011 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.424+0000 7f4c447b2640 1 -- 172.21.15.119:0/560885962 shutdown_connections 2024-09-10T18:57:11.011 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.424+0000 7f4c447b2640 1 --2- 172.21.15.119:0/560885962 >> v2:172.21.15.119:6800/1136125279 conn(0x7f4c280393e0 0x7f4c2803b890 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:11.011 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.424+0000 7f4c447b2640 1 --2- 172.21.15.119:0/560885962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c3c100690 0x7f4c3c10f170 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:11.011 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.424+0000 7f4c447b2640 1 -- 172.21.15.119:0/560885962 >> 172.21.15.119:0/560885962 conn(0x7f4c3c0fbe10 msgr2=0x7f4c3c0fc890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:11.011 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.424+0000 7f4c447b2640 1 -- 172.21.15.119:0/560885962 shutdown_connections 2024-09-10T18:57:11.012 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:10.424+0000 7f4c447b2640 1 -- 172.21.15.119:0/560885962 wait complete. 2024-09-10T18:57:11.256 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:11 smithi119 ceph-mon[29577]: from='mgr.14118 172.21.15.119:0/2740329120' entity='mgr.y' 2024-09-10T18:57:11.256 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/560885962' entity='client.admin' 2024-09-10T18:57:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/127317708' entity='client.admin' 2024-09-10T18:57:12.985 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.460+0000 7fa7b6bed640 1 Processor -- start 2024-09-10T18:57:12.985 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.460+0000 7fa7b6bed640 1 -- start start 2024-09-10T18:57:12.985 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.461+0000 7fa7b6bed640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b00ff440 0x7fa7b00ff850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:12.985 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.461+0000 7fa7b6bed640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b00ffd90 con 0x7fa7b00ff440 2024-09-10T18:57:12.985 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.461+0000 7fa7b4989640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b00ff440 0x7fa7b00ff850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:12.985 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.461+0000 7fa7b4989640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b00ff440 0x7fa7b00ff850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36692/0 (socket says 172.21.15.119:36692) 2024-09-10T18:57:12.985 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.461+0000 7fa7b4989640 1 -- 172.21.15.119:0/1556615138 learned_addr learned my addr 172.21.15.119:0/1556615138 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.461+0000 7fa7b4989640 1 -- 172.21.15.119:0/1556615138 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b00ffed0 con 0x7fa7b00ff440 2024-09-10T18:57:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.462+0000 7fa7b4989640 1 --2- 172.21.15.119:0/1556615138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b00ff440 0x7fa7b00ff850 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4009840 tx=0x7fa7a40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d1d28f1b67536c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.462+0000 7fa7af7fe640 1 -- 172.21.15.119:0/1556615138 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7a402add0 con 0x7fa7b00ff440 2024-09-10T18:57:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.462+0000 7fa7af7fe640 1 -- 172.21.15.119:0/1556615138 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fa7a40328b0 con 0x7fa7b00ff440 2024-09-10T18:57:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.462+0000 7fa7af7fe640 1 -- 172.21.15.119:0/1556615138 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7a4031690 con 0x7fa7b00ff440 2024-09-10T18:57:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.463+0000 7fa7b6bed640 1 -- 172.21.15.119:0/1556615138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b00ff440 msgr2=0x7fa7b00ff850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.463+0000 7fa7b6bed640 1 --2- 172.21.15.119:0/1556615138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b00ff440 0x7fa7b00ff850 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4009840 tx=0x7fa7a40091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.463+0000 7fa7b6bed640 1 -- 172.21.15.119:0/1556615138 shutdown_connections 2024-09-10T18:57:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.463+0000 7fa7b6bed640 1 --2- 172.21.15.119:0/1556615138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b00ff440 0x7fa7b00ff850 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.463+0000 7fa7b6bed640 1 -- 172.21.15.119:0/1556615138 >> 172.21.15.119:0/1556615138 conn(0x7fa7b00fabc0 msgr2=0x7fa7b00fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.463+0000 7fa7b6bed640 1 -- 172.21.15.119:0/1556615138 shutdown_connections 2024-09-10T18:57:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.463+0000 7fa7b6bed640 1 -- 172.21.15.119:0/1556615138 wait complete. 2024-09-10T18:57:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.463+0000 7fa7b6bed640 1 Processor -- start 2024-09-10T18:57:12.986 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.463+0000 7fa7b6bed640 1 -- start start 2024-09-10T18:57:12.987 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.464+0000 7fa7b6bed640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b00ff440 0x7fa7b010d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:12.987 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.464+0000 7fa7b6bed640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b010d5e0 con 0x7fa7b00ff440 2024-09-10T18:57:12.987 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.464+0000 7fa7b4989640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b00ff440 0x7fa7b010d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:12.987 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.464+0000 7fa7b4989640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b00ff440 0x7fa7b010d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36694/0 (socket says 172.21.15.119:36694) 2024-09-10T18:57:12.987 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.464+0000 7fa7b4989640 1 -- 172.21.15.119:0/127317708 learned_addr learned my addr 172.21.15.119:0/127317708 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:12.987 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.464+0000 7fa7b4989640 1 -- 172.21.15.119:0/127317708 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a4009450 con 0x7fa7b00ff440 2024-09-10T18:57:12.987 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.464+0000 7fa7b4989640 1 --2- 172.21.15.119:0/127317708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b00ff440 0x7fa7b010d0a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4031450 tx=0x7fa7a4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:12.987 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.464+0000 7fa7ad7fa640 1 -- 172.21.15.119:0/127317708 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7a4031ca0 con 0x7fa7b00ff440 2024-09-10T18:57:12.987 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.464+0000 7fa7ad7fa640 1 -- 172.21.15.119:0/127317708 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fa7a4031e00 con 0x7fa7b00ff440 2024-09-10T18:57:12.987 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.464+0000 7fa7b6bed640 1 -- 172.21.15.119:0/127317708 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b010d7e0 con 0x7fa7b00ff440 2024-09-10T18:57:12.987 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.464+0000 7fa7ad7fa640 1 -- 172.21.15.119:0/127317708 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa7a403a540 con 0x7fa7b00ff440 2024-09-10T18:57:12.988 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.464+0000 7fa7b6bed640 1 -- 172.21.15.119:0/127317708 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b010dc50 con 0x7fa7b00ff440 2024-09-10T18:57:12.988 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.465+0000 7fa7ad7fa640 1 -- 172.21.15.119:0/127317708 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 45457+0+0 (secure 0 0 0) 0x7fa7a4039070 con 0x7fa7b00ff440 2024-09-10T18:57:12.988 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.465+0000 7fa7ad7fa640 1 --2- 172.21.15.119:0/127317708 >> v2:172.21.15.119:6800/1136125279 conn(0x7fa7a003b6c0 0x7fa7a003db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:12.988 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.465+0000 7fa7b6bed640 1 -- 172.21.15.119:0/127317708 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b004b720 con 0x7fa7b00ff440 2024-09-10T18:57:12.988 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.465+0000 7fa7ad7fa640 1 -- 172.21.15.119:0/127317708 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fa7a406c7c0 con 0x7fa7b00ff440 2024-09-10T18:57:12.989 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.465+0000 7fa7affff640 1 --2- 172.21.15.119:0/127317708 >> v2:172.21.15.119:6800/1136125279 conn(0x7fa7a003b6c0 0x7fa7a003db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:12.989 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.466+0000 7fa7affff640 1 --2- 172.21.15.119:0/127317708 >> v2:172.21.15.119:6800/1136125279 conn(0x7fa7a003b6c0 0x7fa7a003db70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa79c006fd0 tx=0x7fa79c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:12.989 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.471+0000 7fa7ad7fa640 1 -- 172.21.15.119:0/127317708 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fa7a403f070 con 0x7fa7b00ff440 2024-09-10T18:57:12.989 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.635+0000 7fa7b6bed640 1 -- 172.21.15.119:0/127317708 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fa7b01012a0 con 0x7fa7b00ff440 2024-09-10T18:57:12.989 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.640+0000 7fa7ad7fa640 1 -- 172.21.15.119:0/127317708 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fa7a4030940 con 0x7fa7b00ff440 2024-09-10T18:57:12.989 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.643+0000 7fa7b6bed640 1 -- 172.21.15.119:0/127317708 >> v2:172.21.15.119:6800/1136125279 conn(0x7fa7a003b6c0 msgr2=0x7fa7a003db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:12.989 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.643+0000 7fa7b6bed640 1 --2- 172.21.15.119:0/127317708 >> v2:172.21.15.119:6800/1136125279 conn(0x7fa7a003b6c0 0x7fa7a003db70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa79c006fd0 tx=0x7fa79c006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:12.989 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.643+0000 7fa7b6bed640 1 -- 172.21.15.119:0/127317708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b00ff440 msgr2=0x7fa7b010d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:12.989 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.643+0000 7fa7b6bed640 1 --2- 172.21.15.119:0/127317708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b00ff440 0x7fa7b010d0a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4031450 tx=0x7fa7a4008040 comp rx=0 tx=0).stop 2024-09-10T18:57:12.989 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.643+0000 7fa7b6bed640 1 -- 172.21.15.119:0/127317708 shutdown_connections 2024-09-10T18:57:12.989 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.643+0000 7fa7b6bed640 1 --2- 172.21.15.119:0/127317708 >> v2:172.21.15.119:6800/1136125279 conn(0x7fa7a003b6c0 0x7fa7a003db70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:12.989 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.643+0000 7fa7b6bed640 1 --2- 172.21.15.119:0/127317708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b00ff440 0x7fa7b010d0a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:12.989 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.643+0000 7fa7b6bed640 1 -- 172.21.15.119:0/127317708 >> 172.21.15.119:0/127317708 conn(0x7fa7b00fabc0 msgr2=0x7fa7b0103940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:12.989 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.643+0000 7fa7b6bed640 1 -- 172.21.15.119:0/127317708 shutdown_connections 2024-09-10T18:57:12.990 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:11.643+0000 7fa7b6bed640 1 -- 172.21.15.119:0/127317708 wait complete. 2024-09-10T18:57:12.990 INFO:teuthology.orchestra.run.smithi119.stdout:Enabling the dashboard module... 2024-09-10T18:57:13.842 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3954917146' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-10T18:57:13.843 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:13 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ignoring --setuser ceph since I am not root 2024-09-10T18:57:13.843 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:13 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ignoring --setgroup ceph since I am not root 2024-09-10T18:57:13.843 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:13 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:13.785+0000 7f9cb3908ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:57:13.843 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:13 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:13.840+0000 7f9cb3908ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:57:14.172 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:13 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:13.906+0000 7f9cb3908ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:57:14.172 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:14 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:14.034+0000 7f9cb3908ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:57:14.172 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:14 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:14.170+0000 7f9cb3908ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:57:14.311 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.372+0000 7fd346c59640 1 Processor -- start 2024-09-10T18:57:14.317 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.373+0000 7fd346c59640 1 -- start start 2024-09-10T18:57:14.317 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.373+0000 7fd346c59640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3400ff440 0x7fd3400ff850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:14.317 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.373+0000 7fd346c59640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3400ffd90 con 0x7fd3400ff440 2024-09-10T18:57:14.317 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.373+0000 7fd3449f5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3400ff440 0x7fd3400ff850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:14.317 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.373+0000 7fd3449f5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3400ff440 0x7fd3400ff850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46960/0 (socket says 172.21.15.119:46960) 2024-09-10T18:57:14.317 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.373+0000 7fd3449f5640 1 -- 172.21.15.119:0/805087694 learned_addr learned my addr 172.21.15.119:0/805087694 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:14.317 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.373+0000 7fd3449f5640 1 -- 172.21.15.119:0/805087694 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3400ffed0 con 0x7fd3400ff440 2024-09-10T18:57:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.374+0000 7fd3449f5640 1 --2- 172.21.15.119:0/805087694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3400ff440 0x7fd3400ff850 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd334009840 tx=0x7fd3340091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbcc608d5e7f1f3a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.374+0000 7fd33f7fe640 1 -- 172.21.15.119:0/805087694 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd33402add0 con 0x7fd3400ff440 2024-09-10T18:57:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.374+0000 7fd33f7fe640 1 -- 172.21.15.119:0/805087694 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fd3340328b0 con 0x7fd3400ff440 2024-09-10T18:57:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.374+0000 7fd33f7fe640 1 -- 172.21.15.119:0/805087694 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd334031690 con 0x7fd3400ff440 2024-09-10T18:57:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.374+0000 7fd346c59640 1 -- 172.21.15.119:0/805087694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3400ff440 msgr2=0x7fd3400ff850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.374+0000 7fd346c59640 1 --2- 172.21.15.119:0/805087694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3400ff440 0x7fd3400ff850 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd334009840 tx=0x7fd3340091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.375+0000 7fd346c59640 1 -- 172.21.15.119:0/805087694 shutdown_connections 2024-09-10T18:57:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.375+0000 7fd346c59640 1 --2- 172.21.15.119:0/805087694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3400ff440 0x7fd3400ff850 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.375+0000 7fd346c59640 1 -- 172.21.15.119:0/805087694 >> 172.21.15.119:0/805087694 conn(0x7fd3400fabc0 msgr2=0x7fd3400fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.375+0000 7fd346c59640 1 -- 172.21.15.119:0/805087694 shutdown_connections 2024-09-10T18:57:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.375+0000 7fd346c59640 1 -- 172.21.15.119:0/805087694 wait complete. 2024-09-10T18:57:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.375+0000 7fd346c59640 1 Processor -- start 2024-09-10T18:57:14.318 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.375+0000 7fd346c59640 1 -- start start 2024-09-10T18:57:14.319 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.375+0000 7fd346c59640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3400ff440 0x7fd340074b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:14.319 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.375+0000 7fd346c59640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd340111fc0 con 0x7fd3400ff440 2024-09-10T18:57:14.319 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.376+0000 7fd3449f5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3400ff440 0x7fd340074b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:14.319 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.376+0000 7fd3449f5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3400ff440 0x7fd340074b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46968/0 (socket says 172.21.15.119:46968) 2024-09-10T18:57:14.319 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.376+0000 7fd3449f5640 1 -- 172.21.15.119:0/3954917146 learned_addr learned my addr 172.21.15.119:0/3954917146 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:14.319 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.376+0000 7fd3449f5640 1 -- 172.21.15.119:0/3954917146 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd334009450 con 0x7fd3400ff440 2024-09-10T18:57:14.319 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.376+0000 7fd3449f5640 1 --2- 172.21.15.119:0/3954917146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3400ff440 0x7fd340074b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd334006f00 tx=0x7fd334008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:14.319 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.376+0000 7fd33d7fa640 1 -- 172.21.15.119:0/3954917146 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd334031c60 con 0x7fd3400ff440 2024-09-10T18:57:14.319 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.376+0000 7fd33d7fa640 1 -- 172.21.15.119:0/3954917146 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fd334031dc0 con 0x7fd3400ff440 2024-09-10T18:57:14.319 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.376+0000 7fd33d7fa640 1 -- 172.21.15.119:0/3954917146 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd33403a540 con 0x7fd3400ff440 2024-09-10T18:57:14.319 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.376+0000 7fd346c59640 1 -- 172.21.15.119:0/3954917146 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd340112100 con 0x7fd3400ff440 2024-09-10T18:57:14.319 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.376+0000 7fd346c59640 1 -- 172.21.15.119:0/3954917146 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd340112570 con 0x7fd3400ff440 2024-09-10T18:57:14.319 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.377+0000 7fd33d7fa640 1 -- 172.21.15.119:0/3954917146 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 8) v1 ==== 45457+0+0 (secure 0 0 0) 0x7fd334039070 con 0x7fd3400ff440 2024-09-10T18:57:14.320 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.377+0000 7fd33d7fa640 1 --2- 172.21.15.119:0/3954917146 >> v2:172.21.15.119:6800/1136125279 conn(0x7fd330039470 0x7fd33003b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:14.320 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.377+0000 7fd33d7fa640 1 -- 172.21.15.119:0/3954917146 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fd33406c9e0 con 0x7fd3400ff440 2024-09-10T18:57:14.320 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.377+0000 7fd346c59640 1 -- 172.21.15.119:0/3954917146 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34004b720 con 0x7fd3400ff440 2024-09-10T18:57:14.320 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.377+0000 7fd33ffff640 1 --2- 172.21.15.119:0/3954917146 >> v2:172.21.15.119:6800/1136125279 conn(0x7fd330039470 0x7fd33003b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:14.320 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.378+0000 7fd33ffff640 1 --2- 172.21.15.119:0/3954917146 >> v2:172.21.15.119:6800/1136125279 conn(0x7fd330039470 0x7fd33003b920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd32c006fd0 tx=0x7fd32c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:14.320 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.382+0000 7fd33d7fa640 1 -- 172.21.15.119:0/3954917146 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fd33403f080 con 0x7fd3400ff440 2024-09-10T18:57:14.320 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.581+0000 7fd346c59640 1 -- 172.21.15.119:0/3954917146 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fd340112700 con 0x7fd3400ff440 2024-09-10T18:57:14.320 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.646+0000 7fd33d7fa640 1 -- 172.21.15.119:0/3954917146 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 9) v1 ==== 45470+0+0 (secure 0 0 0) 0x7fd33403b960 con 0x7fd3400ff440 2024-09-10T18:57:14.320 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.647+0000 7fd33d7fa640 1 -- 172.21.15.119:0/3954917146 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fd3340449b0 con 0x7fd3400ff440 2024-09-10T18:57:14.320 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.649+0000 7fd346c59640 1 -- 172.21.15.119:0/3954917146 >> v2:172.21.15.119:6800/1136125279 conn(0x7fd330039470 msgr2=0x7fd33003b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:14.320 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.649+0000 7fd346c59640 1 --2- 172.21.15.119:0/3954917146 >> v2:172.21.15.119:6800/1136125279 conn(0x7fd330039470 0x7fd33003b920 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd32c006fd0 tx=0x7fd32c006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:14.320 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.649+0000 7fd346c59640 1 -- 172.21.15.119:0/3954917146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3400ff440 msgr2=0x7fd340074b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:14.321 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.649+0000 7fd346c59640 1 --2- 172.21.15.119:0/3954917146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3400ff440 0x7fd340074b20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd334006f00 tx=0x7fd334008040 comp rx=0 tx=0).stop 2024-09-10T18:57:14.321 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.649+0000 7fd346c59640 1 -- 172.21.15.119:0/3954917146 shutdown_connections 2024-09-10T18:57:14.321 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.649+0000 7fd346c59640 1 --2- 172.21.15.119:0/3954917146 >> v2:172.21.15.119:6800/1136125279 conn(0x7fd330039470 0x7fd33003b920 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:14.321 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.649+0000 7fd346c59640 1 --2- 172.21.15.119:0/3954917146 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3400ff440 0x7fd340074b20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:14.321 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.649+0000 7fd346c59640 1 -- 172.21.15.119:0/3954917146 >> 172.21.15.119:0/3954917146 conn(0x7fd3400fabc0 msgr2=0x7fd3400fb730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:14.321 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.649+0000 7fd346c59640 1 -- 172.21.15.119:0/3954917146 shutdown_connections 2024-09-10T18:57:14.321 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:13.649+0000 7fd346c59640 1 -- 172.21.15.119:0/3954917146 wait complete. 2024-09-10T18:57:14.728 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3954917146' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-10T18:57:14.728 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:14 smithi119 ceph-mon[29577]: mgrmap e9: y(active, since 18s) 2024-09-10T18:57:14.729 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:14 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:14.443+0000 7f9cb3908ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:57:14.729 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:14 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:14.505+0000 7f9cb3908ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:57:14.729 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:14 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:14.674+0000 7f9cb3908ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:57:15.641 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:15 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:15.539+0000 7f9cb3908ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:57:15.644 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:15 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:15.603+0000 7f9cb3908ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:57:15.652 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:57:15.652 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-10T18:57:15.653 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T18:57:15.653 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-10T18:57:15.653 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-10T18:57:15.653 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:57:15.653 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.803+0000 7f9e100ac640 1 Processor -- start 2024-09-10T18:57:15.653 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.803+0000 7f9e100ac640 1 -- start start 2024-09-10T18:57:15.654 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.803+0000 7f9e100ac640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e080ff540 0x7f9e080ff950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:15.654 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.803+0000 7f9e100ac640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080ffe90 con 0x7f9e080ff540 2024-09-10T18:57:15.654 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.803+0000 7f9e0de48640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e080ff540 0x7f9e080ff950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:15.654 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.804+0000 7f9e0de48640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e080ff540 0x7f9e080ff950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46980/0 (socket says 172.21.15.119:46980) 2024-09-10T18:57:15.654 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.804+0000 7f9e0de48640 1 -- 172.21.15.119:0/2613154892 learned_addr learned my addr 172.21.15.119:0/2613154892 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:15.654 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.804+0000 7f9e0de48640 1 -- 172.21.15.119:0/2613154892 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e080fffd0 con 0x7f9e080ff540 2024-09-10T18:57:15.654 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.804+0000 7f9e0de48640 1 --2- 172.21.15.119:0/2613154892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e080ff540 0x7f9e080ff950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9df8009840 tx=0x7f9df80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f3501dd0547f8ba server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:15.654 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.805+0000 7f9e0ce46640 1 -- 172.21.15.119:0/2613154892 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9df802add0 con 0x7f9e080ff540 2024-09-10T18:57:15.654 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.805+0000 7f9e0ce46640 1 -- 172.21.15.119:0/2613154892 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f9df80328b0 con 0x7f9e080ff540 2024-09-10T18:57:15.654 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.805+0000 7f9e0ce46640 1 -- 172.21.15.119:0/2613154892 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9df8031690 con 0x7f9e080ff540 2024-09-10T18:57:15.655 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.805+0000 7f9e100ac640 1 -- 172.21.15.119:0/2613154892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e080ff540 msgr2=0x7f9e080ff950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:15.655 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.805+0000 7f9e100ac640 1 --2- 172.21.15.119:0/2613154892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e080ff540 0x7f9e080ff950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9df8009840 tx=0x7f9df80091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:15.655 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.805+0000 7f9e100ac640 1 -- 172.21.15.119:0/2613154892 shutdown_connections 2024-09-10T18:57:15.655 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.805+0000 7f9e100ac640 1 --2- 172.21.15.119:0/2613154892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e080ff540 0x7f9e080ff950 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:15.655 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.805+0000 7f9e100ac640 1 -- 172.21.15.119:0/2613154892 >> 172.21.15.119:0/2613154892 conn(0x7f9e080fad60 msgr2=0x7f9e080fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:15.655 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.805+0000 7f9e100ac640 1 -- 172.21.15.119:0/2613154892 shutdown_connections 2024-09-10T18:57:15.655 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.805+0000 7f9e100ac640 1 -- 172.21.15.119:0/2613154892 wait complete. 2024-09-10T18:57:15.655 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.806+0000 7f9e100ac640 1 Processor -- start 2024-09-10T18:57:15.655 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.806+0000 7f9e100ac640 1 -- start start 2024-09-10T18:57:15.655 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.806+0000 7f9e100ac640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e080ff540 0x7f9e080753f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:15.656 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.806+0000 7f9e100ac640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08111fd0 con 0x7f9e080ff540 2024-09-10T18:57:15.656 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.806+0000 7f9e0de48640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e080ff540 0x7f9e080753f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:15.656 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.806+0000 7f9e0de48640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e080ff540 0x7f9e080753f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46996/0 (socket says 172.21.15.119:46996) 2024-09-10T18:57:15.656 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.806+0000 7f9e0de48640 1 -- 172.21.15.119:0/1725564770 learned_addr learned my addr 172.21.15.119:0/1725564770 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:15.656 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.807+0000 7f9e0de48640 1 -- 172.21.15.119:0/1725564770 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df8009450 con 0x7f9e080ff540 2024-09-10T18:57:15.656 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.807+0000 7f9e0de48640 1 --2- 172.21.15.119:0/1725564770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e080ff540 0x7f9e080753f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9df8031450 tx=0x7f9df8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:15.656 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.807+0000 7f9dfe7fc640 1 -- 172.21.15.119:0/1725564770 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9df8031ca0 con 0x7f9e080ff540 2024-09-10T18:57:15.656 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.807+0000 7f9dfe7fc640 1 -- 172.21.15.119:0/1725564770 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f9df8031e00 con 0x7f9e080ff540 2024-09-10T18:57:15.656 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.807+0000 7f9e100ac640 1 -- 172.21.15.119:0/1725564770 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e08112110 con 0x7f9e080ff540 2024-09-10T18:57:15.656 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.807+0000 7f9dfe7fc640 1 -- 172.21.15.119:0/1725564770 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9df803a540 con 0x7f9e080ff540 2024-09-10T18:57:15.656 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.807+0000 7f9e100ac640 1 -- 172.21.15.119:0/1725564770 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e08112580 con 0x7f9e080ff540 2024-09-10T18:57:15.657 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.808+0000 7f9dfe7fc640 1 -- 172.21.15.119:0/1725564770 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 9) v1 ==== 45470+0+0 (secure 0 0 0) 0x7f9df8039070 con 0x7f9e080ff540 2024-09-10T18:57:15.657 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.808+0000 7f9e100ac640 1 -- 172.21.15.119:0/1725564770 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0804b720 con 0x7f9e080ff540 2024-09-10T18:57:15.657 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.808+0000 7f9dfe7fc640 1 --2- 172.21.15.119:0/1725564770 >> v2:172.21.15.119:6800/1136125279 conn(0x7f9df40394c0 0x7f9df403b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:15.657 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.808+0000 7f9dfe7fc640 1 -- 172.21.15.119:0/1725564770 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f9df806c9b0 con 0x7f9e080ff540 2024-09-10T18:57:15.657 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.808+0000 7f9e0d647640 1 -- 172.21.15.119:0/1725564770 >> v2:172.21.15.119:6800/1136125279 conn(0x7f9df40394c0 msgr2=0x7f9df403b970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1136125279 2024-09-10T18:57:15.657 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.808+0000 7f9e0d647640 1 --2- 172.21.15.119:0/1725564770 >> v2:172.21.15.119:6800/1136125279 conn(0x7f9df40394c0 0x7f9df403b970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T18:57:15.658 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:14.813+0000 7f9dfe7fc640 1 -- 172.21.15.119:0/1725564770 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f9df8030b20 con 0x7f9e080ff540 2024-09-10T18:57:15.658 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:15.009+0000 7f9e0d647640 1 -- 172.21.15.119:0/1725564770 >> v2:172.21.15.119:6800/1136125279 conn(0x7f9df40394c0 msgr2=0x7f9df403b970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1136125279 2024-09-10T18:57:15.658 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:15.009+0000 7f9e0d647640 1 --2- 172.21.15.119:0/1725564770 >> v2:172.21.15.119:6800/1136125279 conn(0x7f9df40394c0 0x7f9df403b970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T18:57:15.658 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:15.018+0000 7f9e100ac640 1 -- 172.21.15.119:0/1725564770 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9e08112710 con 0x7f9e080ff540 2024-09-10T18:57:15.658 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:15.018+0000 7f9dfe7fc640 1 -- 172.21.15.119:0/1725564770 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f9df803f020 con 0x7f9e080ff540 2024-09-10T18:57:15.659 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:15.020+0000 7f9e100ac640 1 -- 172.21.15.119:0/1725564770 >> v2:172.21.15.119:6800/1136125279 conn(0x7f9df40394c0 msgr2=0x7f9df403b970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T18:57:15.659 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:15.020+0000 7f9e100ac640 1 --2- 172.21.15.119:0/1725564770 >> v2:172.21.15.119:6800/1136125279 conn(0x7f9df40394c0 0x7f9df403b970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:15.659 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:15.020+0000 7f9e100ac640 1 -- 172.21.15.119:0/1725564770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e080ff540 msgr2=0x7f9e080753f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:15.659 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:15.020+0000 7f9e100ac640 1 --2- 172.21.15.119:0/1725564770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e080ff540 0x7f9e080753f0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9df8031450 tx=0x7f9df8008040 comp rx=0 tx=0).stop 2024-09-10T18:57:15.659 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:15.020+0000 7f9e100ac640 1 -- 172.21.15.119:0/1725564770 shutdown_connections 2024-09-10T18:57:15.659 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:15.020+0000 7f9e100ac640 1 --2- 172.21.15.119:0/1725564770 >> v2:172.21.15.119:6800/1136125279 conn(0x7f9df40394c0 0x7f9df403b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:15.660 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:15.020+0000 7f9e100ac640 1 --2- 172.21.15.119:0/1725564770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e080ff540 0x7f9e080753f0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:15.660 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:15.020+0000 7f9e100ac640 1 -- 172.21.15.119:0/1725564770 >> 172.21.15.119:0/1725564770 conn(0x7f9e080fad60 msgr2=0x7f9e080fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:15.660 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:15.020+0000 7f9e100ac640 1 -- 172.21.15.119:0/1725564770 shutdown_connections 2024-09-10T18:57:15.660 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:15.020+0000 7f9e100ac640 1 -- 172.21.15.119:0/1725564770 wait complete. 2024-09-10T18:57:15.660 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for the mgr to restart... 2024-09-10T18:57:15.660 INFO:teuthology.orchestra.run.smithi119.stdout:Waiting for mgr epoch 9... 2024-09-10T18:57:15.902 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1725564770' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-10T18:57:16.581 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:16 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:16.348+0000 7f9cb3908ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:57:16.582 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:16 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:16.459+0000 7f9cb3908ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:57:16.834 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:16 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:16.580+0000 7f9cb3908ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:57:16.834 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:16 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:16.639+0000 7f9cb3908ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:57:16.834 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:16 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:16.765+0000 7f9cb3908ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:57:16.834 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:16 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:16.832+0000 7f9cb3908ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:57:17.089 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:16 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:16.890+0000 7f9cb3908ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:57:17.089 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:17 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:17.024+0000 7f9cb3908ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:57:17.089 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:17 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:17.087+0000 7f9cb3908ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:57:17.348 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:17 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:17.155+0000 7f9cb3908ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:57:17.349 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:17 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:17.236+0000 7f9cb3908ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:57:17.349 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:17 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:17.296+0000 7f9cb3908ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:57:17.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:17 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:17.355+0000 7f9cb3908ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:57:18.566 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:18.126+0000 7f9cb3908ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:57:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:18 smithi119 ceph-mon[29577]: Active manager daemon y restarted 2024-09-10T18:57:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:18 smithi119 ceph-mon[29577]: Activating manager daemon y 2024-09-10T18:57:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:18 smithi119 ceph-mon[29577]: osdmap e3: 0 total, 0 up, 0 in 2024-09-10T18:57:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:18 smithi119 ceph-mon[29577]: mgrmap e10: y(active, starting, since 0.00331911s) 2024-09-10T18:57:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:57:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T18:57:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T18:57:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T18:57:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T18:57:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:18 smithi119 ceph-mon[29577]: Manager daemon y is now available 2024-09-10T18:57:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:57:18.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T18:57:18.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T18:57:19.860 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout { 2024-09-10T18:57:19.860 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-10T18:57:19.860 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-10T18:57:19.860 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout } 2024-09-10T18:57:19.860 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.119+0000 7f7314987640 1 Processor -- start 2024-09-10T18:57:19.860 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.119+0000 7f7314987640 1 -- start start 2024-09-10T18:57:19.860 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.119+0000 7f7314987640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73100753f0 0x7f7310105670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:19.861 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.119+0000 7f7314987640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7310075800 con 0x7f73100753f0 2024-09-10T18:57:19.861 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.119+0000 7f730e59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73100753f0 0x7f7310105670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:19.861 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.119+0000 7f730e59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73100753f0 0x7f7310105670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47002/0 (socket says 172.21.15.119:47002) 2024-09-10T18:57:19.861 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.119+0000 7f730e59c640 1 -- 172.21.15.119:0/554820995 learned_addr learned my addr 172.21.15.119:0/554820995 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:19.861 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.120+0000 7f730e59c640 1 -- 172.21.15.119:0/554820995 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7310101060 con 0x7f73100753f0 2024-09-10T18:57:19.861 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.120+0000 7f730e59c640 1 --2- 172.21.15.119:0/554820995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73100753f0 0x7f7310105670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f72f8009840 tx=0x7f72f80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14cd6a0fa0a6edf8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:19.861 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.120+0000 7f730d59a640 1 -- 172.21.15.119:0/554820995 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72f802add0 con 0x7f73100753f0 2024-09-10T18:57:19.861 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.121+0000 7f730d59a640 1 -- 172.21.15.119:0/554820995 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f72f80328b0 con 0x7f73100753f0 2024-09-10T18:57:19.861 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.121+0000 7f730d59a640 1 -- 172.21.15.119:0/554820995 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72f8031690 con 0x7f73100753f0 2024-09-10T18:57:19.861 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.121+0000 7f7314987640 1 -- 172.21.15.119:0/554820995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73100753f0 msgr2=0x7f7310105670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:19.861 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.121+0000 7f7314987640 1 --2- 172.21.15.119:0/554820995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73100753f0 0x7f7310105670 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f72f8009840 tx=0x7f72f80091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:19.862 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.121+0000 7f7314987640 1 -- 172.21.15.119:0/554820995 shutdown_connections 2024-09-10T18:57:19.862 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.121+0000 7f7314987640 1 --2- 172.21.15.119:0/554820995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73100753f0 0x7f7310105670 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:19.862 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.121+0000 7f7314987640 1 -- 172.21.15.119:0/554820995 >> 172.21.15.119:0/554820995 conn(0x7f73100facc0 msgr2=0x7f73100fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:19.862 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.122+0000 7f7314987640 1 -- 172.21.15.119:0/554820995 shutdown_connections 2024-09-10T18:57:19.862 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.122+0000 7f7314987640 1 -- 172.21.15.119:0/554820995 wait complete. 2024-09-10T18:57:19.862 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.122+0000 7f7314987640 1 Processor -- start 2024-09-10T18:57:19.862 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.122+0000 7f7314987640 1 -- start start 2024-09-10T18:57:19.863 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.122+0000 7f7314987640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73100753f0 0x7f7310111460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:19.863 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.122+0000 7f7314987640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73101119a0 con 0x7f73100753f0 2024-09-10T18:57:19.863 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.123+0000 7f730e59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73100753f0 0x7f7310111460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:19.863 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.123+0000 7f730e59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73100753f0 0x7f7310111460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47018/0 (socket says 172.21.15.119:47018) 2024-09-10T18:57:19.863 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.123+0000 7f730e59c640 1 -- 172.21.15.119:0/330523478 learned_addr learned my addr 172.21.15.119:0/330523478 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:19.863 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.123+0000 7f730e59c640 1 -- 172.21.15.119:0/330523478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f8009450 con 0x7f73100753f0 2024-09-10T18:57:19.863 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.123+0000 7f730e59c640 1 --2- 172.21.15.119:0/330523478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73100753f0 0x7f7310111460 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72f8006f00 tx=0x7f72f8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:19.863 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.123+0000 7f7306ffd640 1 -- 172.21.15.119:0/330523478 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72f8031c60 con 0x7f73100753f0 2024-09-10T18:57:19.863 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.123+0000 7f7306ffd640 1 -- 172.21.15.119:0/330523478 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f72f8031dc0 con 0x7f73100753f0 2024-09-10T18:57:19.864 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.123+0000 7f7314987640 1 -- 172.21.15.119:0/330523478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7310111ba0 con 0x7f73100753f0 2024-09-10T18:57:19.864 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.123+0000 7f7306ffd640 1 -- 172.21.15.119:0/330523478 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72f803a540 con 0x7f73100753f0 2024-09-10T18:57:19.864 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.123+0000 7f7314987640 1 -- 172.21.15.119:0/330523478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7310112730 con 0x7f73100753f0 2024-09-10T18:57:19.864 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.124+0000 7f7306ffd640 1 -- 172.21.15.119:0/330523478 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 9) v1 ==== 45470+0+0 (secure 0 0 0) 0x7f72f8039070 con 0x7f73100753f0 2024-09-10T18:57:19.864 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.124+0000 7f7306ffd640 1 --2- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1136125279 conn(0x7f72fc0394c0 0x7f72fc03b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:19.864 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.124+0000 7f7314987640 1 -- 172.21.15.119:0/330523478 --> v2:172.21.15.119:6800/1136125279 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f731005ed70 con 0x7f72fc0394c0 2024-09-10T18:57:19.864 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.124+0000 7f730dd9b640 1 -- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1136125279 conn(0x7f72fc0394c0 msgr2=0x7f72fc03b970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1136125279 2024-09-10T18:57:19.864 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.124+0000 7f7306ffd640 1 -- 172.21.15.119:0/330523478 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f72f806cb40 con 0x7f73100753f0 2024-09-10T18:57:19.865 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.124+0000 7f730dd9b640 1 --2- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1136125279 conn(0x7f72fc0394c0 0x7f72fc03b970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T18:57:19.865 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.325+0000 7f730dd9b640 1 -- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1136125279 conn(0x7f72fc0394c0 msgr2=0x7f72fc03b970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1136125279 2024-09-10T18:57:19.865 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.325+0000 7f730dd9b640 1 --2- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1136125279 conn(0x7f72fc0394c0 0x7f72fc03b970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T18:57:19.865 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.726+0000 7f730dd9b640 1 -- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1136125279 conn(0x7f72fc0394c0 msgr2=0x7f72fc03b970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1136125279 2024-09-10T18:57:19.865 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:16.726+0000 7f730dd9b640 1 --2- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1136125279 conn(0x7f72fc0394c0 0x7f72fc03b970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T18:57:19.865 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:17.527+0000 7f730dd9b640 1 -- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1136125279 conn(0x7f72fc0394c0 msgr2=0x7f72fc03b970 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1136125279 2024-09-10T18:57:19.865 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:17.527+0000 7f730dd9b640 1 --2- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1136125279 conn(0x7f72fc0394c0 0x7f72fc03b970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T18:57:19.865 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:18.130+0000 7f7306ffd640 1 -- 172.21.15.119:0/330523478 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mgrmap(e 10) v1 ==== 45272+0+0 (secure 0 0 0) 0x7f72f8039cf0 con 0x7f73100753f0 2024-09-10T18:57:19.866 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:18.130+0000 7f7306ffd640 1 -- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1136125279 conn(0x7f72fc0394c0 msgr2=0x7f72fc03b970 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T18:57:19.866 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:18.130+0000 7f7306ffd640 1 --2- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1136125279 conn(0x7f72fc0394c0 0x7f72fc03b970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:19.866 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.131+0000 7f7306ffd640 1 -- 172.21.15.119:0/330523478 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mgrmap(e 11) v1 ==== 45364+0+0 (secure 0 0 0) 0x7f72f804ccf0 con 0x7f73100753f0 2024-09-10T18:57:19.866 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.131+0000 7f7306ffd640 1 --2- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1094129222 conn(0x7f72fc03cd20 0x7f72fc03f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:19.866 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.131+0000 7f7306ffd640 1 -- 172.21.15.119:0/330523478 --> v2:172.21.15.119:6800/1094129222 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f731005ed70 con 0x7f72fc03cd20 2024-09-10T18:57:19.866 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.131+0000 7f730dd9b640 1 --2- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1094129222 conn(0x7f72fc03cd20 0x7f72fc03f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:19.867 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.132+0000 7f730dd9b640 1 --2- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1094129222 conn(0x7f72fc03cd20 0x7f72fc03f100 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7300003940 tx=0x7f7300009350 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:19.867 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.133+0000 7f7306ffd640 1 -- 172.21.15.119:0/330523478 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f731005ed70 con 0x7f72fc03cd20 2024-09-10T18:57:19.867 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.138+0000 7f7314987640 1 -- 172.21.15.119:0/330523478 --> v2:172.21.15.119:6800/1094129222 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f7310112070 con 0x7f72fc03cd20 2024-09-10T18:57:19.867 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.138+0000 7f7306ffd640 1 -- 172.21.15.119:0/330523478 <== mgr.14150 v2:172.21.15.119:6800/1094129222 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f7310112070 con 0x7f72fc03cd20 2024-09-10T18:57:19.867 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.138+0000 7f7314987640 1 -- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1094129222 conn(0x7f72fc03cd20 msgr2=0x7f72fc03f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:19.867 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.138+0000 7f7314987640 1 --2- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1094129222 conn(0x7f72fc03cd20 0x7f72fc03f100 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f7300003940 tx=0x7f7300009350 comp rx=0 tx=0).stop 2024-09-10T18:57:19.868 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.138+0000 7f7314987640 1 -- 172.21.15.119:0/330523478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73100753f0 msgr2=0x7f7310111460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:19.868 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.138+0000 7f7314987640 1 --2- 172.21.15.119:0/330523478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73100753f0 0x7f7310111460 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f72f8006f00 tx=0x7f72f8008040 comp rx=0 tx=0).stop 2024-09-10T18:57:19.868 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.138+0000 7f7314987640 1 -- 172.21.15.119:0/330523478 shutdown_connections 2024-09-10T18:57:19.868 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.138+0000 7f7314987640 1 --2- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1094129222 conn(0x7f72fc03cd20 0x7f72fc03f100 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:19.868 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.138+0000 7f7314987640 1 --2- 172.21.15.119:0/330523478 >> v2:172.21.15.119:6800/1136125279 conn(0x7f72fc0394c0 0x7f72fc03b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:19.868 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.138+0000 7f7314987640 1 --2- 172.21.15.119:0/330523478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73100753f0 0x7f7310111460 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:19.868 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.138+0000 7f7314987640 1 -- 172.21.15.119:0/330523478 >> 172.21.15.119:0/330523478 conn(0x7f73100facc0 msgr2=0x7f73100fb800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:19.869 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.138+0000 7f7314987640 1 -- 172.21.15.119:0/330523478 shutdown_connections 2024-09-10T18:57:19.869 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:19.138+0000 7f7314987640 1 -- 172.21.15.119:0/330523478 wait complete. 2024-09-10T18:57:19.869 INFO:teuthology.orchestra.run.smithi119.stdout:mgr epoch 9 is available 2024-09-10T18:57:19.869 INFO:teuthology.orchestra.run.smithi119.stdout:Generating a dashboard self-signed certificate... 2024-09-10T18:57:20.133 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:19.942+0000 7f9c64465640 -1 log_channel(cephadm) log [ERR] : [10/Sep/2024:18:57:19] ENGINE Error in HTTPServer.serve 2024-09-10T18:57:20.133 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: Traceback (most recent call last): 2024-09-10T18:57:20.133 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:57:20.133 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: self._connections.run(self.expiration_interval) 2024-09-10T18:57:20.133 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:57:20.133 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: self._run(expiration_interval) 2024-09-10T18:57:20.133 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:57:20.133 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:57:20.133 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:57:20.133 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:57:20.133 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:57:20.133 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: s = self.context.wrap_socket( 2024-09-10T18:57:20.134 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:57:20.134 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: return self.sslsocket_class._create( 2024-09-10T18:57:20.134 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:57:20.134 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: self.do_handshake() 2024-09-10T18:57:20.134 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:57:20.134 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: self._sslobj.do_handshake() 2024-09-10T18:57:20.134 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:57:20.134 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 18:57:19 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T18:57:20.491 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:20 smithi119 ceph-mon[29577]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-10T18:57:20.491 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:20 smithi119 ceph-mon[29577]: mgrmap e11: y(active, since 1.0056s) 2024-09-10T18:57:20.491 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:20 smithi119 ceph-mon[29577]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-10T18:57:20.491 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:20 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:21.194 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-10T18:57:21.194 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.312+0000 7ff7f76a8640 1 Processor -- start 2024-09-10T18:57:21.194 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.312+0000 7ff7f76a8640 1 -- start start 2024-09-10T18:57:21.194 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.313+0000 7ff7f76a8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7f0100670 0x7ff7f0100a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:21.194 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.313+0000 7ff7f76a8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f0100fc0 con 0x7ff7f0100670 2024-09-10T18:57:21.194 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.313+0000 7ff7f5444640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7f0100670 0x7ff7f0100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:21.194 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.313+0000 7ff7f5444640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7f0100670 0x7ff7f0100a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47090/0 (socket says 172.21.15.119:47090) 2024-09-10T18:57:21.194 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.313+0000 7ff7f5444640 1 -- 172.21.15.119:0/1218532696 learned_addr learned my addr 172.21.15.119:0/1218532696 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:21.195 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.313+0000 7ff7f5444640 1 -- 172.21.15.119:0/1218532696 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f0101100 con 0x7ff7f0100670 2024-09-10T18:57:21.195 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.314+0000 7ff7f5444640 1 --2- 172.21.15.119:0/1218532696 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7f0100670 0x7ff7f0100a80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009840 tx=0x7ff7ec0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1bafec770dc66 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:21.195 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.314+0000 7ff7e7fff640 1 -- 172.21.15.119:0/1218532696 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7ec02add0 con 0x7ff7f0100670 2024-09-10T18:57:21.195 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.314+0000 7ff7e7fff640 1 -- 172.21.15.119:0/1218532696 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7ff7ec0328b0 con 0x7ff7f0100670 2024-09-10T18:57:21.195 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.314+0000 7ff7e7fff640 1 -- 172.21.15.119:0/1218532696 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7ec031690 con 0x7ff7f0100670 2024-09-10T18:57:21.195 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.315+0000 7ff7f76a8640 1 -- 172.21.15.119:0/1218532696 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7f0100670 msgr2=0x7ff7f0100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:21.195 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.315+0000 7ff7f76a8640 1 --2- 172.21.15.119:0/1218532696 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7f0100670 0x7ff7f0100a80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec009840 tx=0x7ff7ec0091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:21.195 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.315+0000 7ff7f76a8640 1 -- 172.21.15.119:0/1218532696 shutdown_connections 2024-09-10T18:57:21.195 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.315+0000 7ff7f76a8640 1 --2- 172.21.15.119:0/1218532696 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7f0100670 0x7ff7f0100a80 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:21.195 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.315+0000 7ff7f76a8640 1 -- 172.21.15.119:0/1218532696 >> 172.21.15.119:0/1218532696 conn(0x7ff7f00fbe10 msgr2=0x7ff7f00fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:21.195 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.315+0000 7ff7f76a8640 1 -- 172.21.15.119:0/1218532696 shutdown_connections 2024-09-10T18:57:21.195 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.315+0000 7ff7f76a8640 1 -- 172.21.15.119:0/1218532696 wait complete. 2024-09-10T18:57:21.196 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.315+0000 7ff7f76a8640 1 Processor -- start 2024-09-10T18:57:21.196 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.315+0000 7ff7f76a8640 1 -- start start 2024-09-10T18:57:21.196 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.315+0000 7ff7f76a8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7f0100670 0x7ff7f010f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:21.196 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.315+0000 7ff7f76a8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7f010f830 con 0x7ff7f0100670 2024-09-10T18:57:21.196 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.316+0000 7ff7f5444640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7f0100670 0x7ff7f010f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:21.196 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.316+0000 7ff7f5444640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7f0100670 0x7ff7f010f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47104/0 (socket says 172.21.15.119:47104) 2024-09-10T18:57:21.196 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.316+0000 7ff7f5444640 1 -- 172.21.15.119:0/165410412 learned_addr learned my addr 172.21.15.119:0/165410412 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:21.196 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.316+0000 7ff7f5444640 1 -- 172.21.15.119:0/165410412 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ec009450 con 0x7ff7f0100670 2024-09-10T18:57:21.196 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.316+0000 7ff7f5444640 1 --2- 172.21.15.119:0/165410412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7f0100670 0x7ff7f010f2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec031450 tx=0x7ff7ec008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:21.196 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.316+0000 7ff7e5ffb640 1 -- 172.21.15.119:0/165410412 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7ec031ca0 con 0x7ff7f0100670 2024-09-10T18:57:21.196 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.316+0000 7ff7e5ffb640 1 -- 172.21.15.119:0/165410412 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7ff7ec031e00 con 0x7ff7f0100670 2024-09-10T18:57:21.196 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.316+0000 7ff7f76a8640 1 -- 172.21.15.119:0/165410412 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7f010fa30 con 0x7ff7f0100670 2024-09-10T18:57:21.197 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.316+0000 7ff7e5ffb640 1 -- 172.21.15.119:0/165410412 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7ec03a540 con 0x7ff7f0100670 2024-09-10T18:57:21.197 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.316+0000 7ff7f76a8640 1 -- 172.21.15.119:0/165410412 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7f010fea0 con 0x7ff7f0100670 2024-09-10T18:57:21.197 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.317+0000 7ff7e5ffb640 1 -- 172.21.15.119:0/165410412 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 11) v1 ==== 45364+0+0 (secure 0 0 0) 0x7ff7ec039070 con 0x7ff7f0100670 2024-09-10T18:57:21.197 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.317+0000 7ff7f76a8640 1 -- 172.21.15.119:0/165410412 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7f004b720 con 0x7ff7f0100670 2024-09-10T18:57:21.197 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.317+0000 7ff7e5ffb640 1 --2- 172.21.15.119:0/165410412 >> v2:172.21.15.119:6800/1094129222 conn(0x7ff7dc0393b0 0x7ff7dc03b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:21.197 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.317+0000 7ff7e5ffb640 1 -- 172.21.15.119:0/165410412 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff7ec06cb00 con 0x7ff7f0100670 2024-09-10T18:57:21.197 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.317+0000 7ff7f4c43640 1 --2- 172.21.15.119:0/165410412 >> v2:172.21.15.119:6800/1094129222 conn(0x7ff7dc0393b0 0x7ff7dc03b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:21.197 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.318+0000 7ff7f4c43640 1 --2- 172.21.15.119:0/165410412 >> v2:172.21.15.119:6800/1094129222 conn(0x7ff7dc0393b0 0x7ff7dc03b860 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0006fd0 tx=0x7ff7e0006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:21.197 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.322+0000 7ff7e5ffb640 1 -- 172.21.15.119:0/165410412 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7ff7ec03a740 con 0x7ff7f0100670 2024-09-10T18:57:21.197 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.489+0000 7ff7f76a8640 1 -- 172.21.15.119:0/165410412 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7ff7f010fea0 con 0x7ff7dc0393b0 2024-09-10T18:57:21.197 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.571+0000 7ff7e5ffb640 1 -- 172.21.15.119:0/165410412 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7ff7f010fea0 con 0x7ff7dc0393b0 2024-09-10T18:57:21.197 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.573+0000 7ff7f76a8640 1 -- 172.21.15.119:0/165410412 >> v2:172.21.15.119:6800/1094129222 conn(0x7ff7dc0393b0 msgr2=0x7ff7dc03b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:21.198 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.573+0000 7ff7f76a8640 1 --2- 172.21.15.119:0/165410412 >> v2:172.21.15.119:6800/1094129222 conn(0x7ff7dc0393b0 0x7ff7dc03b860 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7ff7e0006fd0 tx=0x7ff7e0006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:21.198 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.573+0000 7ff7f76a8640 1 -- 172.21.15.119:0/165410412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7f0100670 msgr2=0x7ff7f010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:21.198 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.573+0000 7ff7f76a8640 1 --2- 172.21.15.119:0/165410412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7f0100670 0x7ff7f010f2f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff7ec031450 tx=0x7ff7ec008040 comp rx=0 tx=0).stop 2024-09-10T18:57:21.198 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.573+0000 7ff7f76a8640 1 -- 172.21.15.119:0/165410412 shutdown_connections 2024-09-10T18:57:21.198 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.573+0000 7ff7f76a8640 1 --2- 172.21.15.119:0/165410412 >> v2:172.21.15.119:6800/1094129222 conn(0x7ff7dc0393b0 0x7ff7dc03b860 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:21.198 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.573+0000 7ff7f76a8640 1 --2- 172.21.15.119:0/165410412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7f0100670 0x7ff7f010f2f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:21.198 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.573+0000 7ff7f76a8640 1 -- 172.21.15.119:0/165410412 >> 172.21.15.119:0/165410412 conn(0x7ff7f00fbe10 msgr2=0x7ff7f00fc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:21.198 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.573+0000 7ff7f76a8640 1 -- 172.21.15.119:0/165410412 shutdown_connections 2024-09-10T18:57:21.198 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:20.573+0000 7ff7f76a8640 1 -- 172.21.15.119:0/165410412 wait complete. 2024-09-10T18:57:21.198 INFO:teuthology.orchestra.run.smithi119.stdout:Creating initial admin user... 2024-09-10T18:57:21.442 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:21 smithi119 ceph-mon[29577]: [10/Sep/2024:18:57:19] ENGINE Bus STARTING 2024-09-10T18:57:21.442 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:21 smithi119 ceph-mon[29577]: [10/Sep/2024:18:57:19] ENGINE Serving on https://172.21.15.119:7150 2024-09-10T18:57:21.442 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:21 smithi119 ceph-mon[29577]: [10/Sep/2024:18:57:19] ENGINE Bus STARTED 2024-09-10T18:57:21.442 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:21 smithi119 ceph-mon[29577]: [10/Sep/2024:18:57:19] ENGINE Error in HTTPServer.serve 2024-09-10T18:57:21.442 INFO:journalctl@ceph.mon.a.smithi119.stdout: Traceback (most recent call last): 2024-09-10T18:57:21.442 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T18:57:21.442 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._connections.run(self.expiration_interval) 2024-09-10T18:57:21.442 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T18:57:21.442 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._run(expiration_interval) 2024-09-10T18:57:21.442 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T18:57:21.442 INFO:journalctl@ceph.mon.a.smithi119.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T18:57:21.443 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T18:57:21.443 INFO:journalctl@ceph.mon.a.smithi119.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T18:57:21.443 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T18:57:21.443 INFO:journalctl@ceph.mon.a.smithi119.stdout: s = self.context.wrap_socket( 2024-09-10T18:57:21.443 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T18:57:21.443 INFO:journalctl@ceph.mon.a.smithi119.stdout: return self.sslsocket_class._create( 2024-09-10T18:57:21.443 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T18:57:21.443 INFO:journalctl@ceph.mon.a.smithi119.stdout: self.do_handshake() 2024-09-10T18:57:21.443 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T18:57:21.443 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._sslobj.do_handshake() 2024-09-10T18:57:21.443 INFO:journalctl@ceph.mon.a.smithi119.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T18:57:21.443 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:21 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:21.443 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:21 smithi119 ceph-mon[29577]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:57:21.443 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:21 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:21.444 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:21 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:21.444 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:21 smithi119 ceph-mon[29577]: mgrmap e12: y(active, since 2s) 2024-09-10T18:57:21.444 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:21 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:22.908 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$vR2WN0jpFuIVfTWnpP/K.OQw2Bz.qA9VceNvfBdyFYbJnII.ySA6a", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1725994642, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-10T18:57:22.908 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.656+0000 7f7a9ac42640 1 Processor -- start 2024-09-10T18:57:22.908 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.656+0000 7f7a9ac42640 1 -- start start 2024-09-10T18:57:22.908 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.656+0000 7f7a9ac42640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a94075480 0x7f7a941031f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:22.908 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.656+0000 7f7a9ac42640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a94103730 con 0x7f7a94075480 2024-09-10T18:57:22.908 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.656+0000 7f7a989de640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a94075480 0x7f7a941031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:22.908 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.656+0000 7f7a989de640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a94075480 0x7f7a941031f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47118/0 (socket says 172.21.15.119:47118) 2024-09-10T18:57:22.908 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.656+0000 7f7a989de640 1 -- 172.21.15.119:0/2564803739 learned_addr learned my addr 172.21.15.119:0/2564803739 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:22.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.657+0000 7f7a989de640 1 -- 172.21.15.119:0/2564803739 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a94103870 con 0x7f7a94075480 2024-09-10T18:57:22.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.657+0000 7f7a989de640 1 --2- 172.21.15.119:0/2564803739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a94075480 0x7f7a941031f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009840 tx=0x7f7a880091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a083f189eae1fb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:22.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.658+0000 7f7a937fe640 1 -- 172.21.15.119:0/2564803739 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a8802add0 con 0x7f7a94075480 2024-09-10T18:57:22.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.658+0000 7f7a937fe640 1 -- 172.21.15.119:0/2564803739 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f7a880328b0 con 0x7f7a94075480 2024-09-10T18:57:22.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.658+0000 7f7a937fe640 1 -- 172.21.15.119:0/2564803739 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a88031690 con 0x7f7a94075480 2024-09-10T18:57:22.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.658+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2564803739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a94075480 msgr2=0x7f7a941031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:22.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.658+0000 7f7a9ac42640 1 --2- 172.21.15.119:0/2564803739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a94075480 0x7f7a941031f0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f7a88009840 tx=0x7f7a880091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:22.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.658+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2564803739 shutdown_connections 2024-09-10T18:57:22.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.658+0000 7f7a9ac42640 1 --2- 172.21.15.119:0/2564803739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a94075480 0x7f7a941031f0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:22.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.658+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2564803739 >> 172.21.15.119:0/2564803739 conn(0x7f7a940facc0 msgr2=0x7f7a940fd0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:22.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.658+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2564803739 shutdown_connections 2024-09-10T18:57:22.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.658+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2564803739 wait complete. 2024-09-10T18:57:22.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.659+0000 7f7a9ac42640 1 Processor -- start 2024-09-10T18:57:22.909 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.659+0000 7f7a9ac42640 1 -- start start 2024-09-10T18:57:22.910 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.659+0000 7f7a9ac42640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a94075480 0x7f7a9410f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:22.910 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.659+0000 7f7a9ac42640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9410f7f0 con 0x7f7a94075480 2024-09-10T18:57:22.910 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.660+0000 7f7a989de640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a94075480 0x7f7a9410f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:22.910 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.660+0000 7f7a989de640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a94075480 0x7f7a9410f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47126/0 (socket says 172.21.15.119:47126) 2024-09-10T18:57:22.910 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.660+0000 7f7a989de640 1 -- 172.21.15.119:0/2759266635 learned_addr learned my addr 172.21.15.119:0/2759266635 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:22.910 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.660+0000 7f7a989de640 1 -- 172.21.15.119:0/2759266635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a88009450 con 0x7f7a94075480 2024-09-10T18:57:22.910 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.660+0000 7f7a989de640 1 --2- 172.21.15.119:0/2759266635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a94075480 0x7f7a9410f2b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7a88000c00 tx=0x7f7a88008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:22.910 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.660+0000 7f7a917fa640 1 -- 172.21.15.119:0/2759266635 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a88031c50 con 0x7f7a94075480 2024-09-10T18:57:22.910 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.660+0000 7f7a917fa640 1 -- 172.21.15.119:0/2759266635 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f7a88031db0 con 0x7f7a94075480 2024-09-10T18:57:22.910 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.660+0000 7f7a917fa640 1 -- 172.21.15.119:0/2759266635 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a8803a3a0 con 0x7f7a94075480 2024-09-10T18:57:22.910 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.660+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2759266635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a9410f9f0 con 0x7f7a94075480 2024-09-10T18:57:22.910 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.660+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2759266635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9410fe60 con 0x7f7a94075480 2024-09-10T18:57:22.910 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.661+0000 7f7a917fa640 1 -- 172.21.15.119:0/2759266635 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 12) v1 ==== 45470+0+0 (secure 0 0 0) 0x7f7a88039070 con 0x7f7a94075480 2024-09-10T18:57:22.910 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.661+0000 7f7a917fa640 1 --2- 172.21.15.119:0/2759266635 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7a84039470 0x7f7a8403b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:22.911 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.661+0000 7f7a917fa640 1 -- 172.21.15.119:0/2759266635 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7a8806d610 con 0x7f7a94075480 2024-09-10T18:57:22.911 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.662+0000 7f7a93fff640 1 --2- 172.21.15.119:0/2759266635 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7a84039470 0x7f7a8403b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:22.911 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.662+0000 7f7a93fff640 1 --2- 172.21.15.119:0/2759266635 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7a84039470 0x7f7a8403b920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a80006fd0 tx=0x7f7a80006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:22.911 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.662+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2759266635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a9410fe60 con 0x7f7a94075480 2024-09-10T18:57:22.911 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.667+0000 7f7a917fa640 1 -- 172.21.15.119:0/2759266635 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f7a8803f070 con 0x7f7a94075480 2024-09-10T18:57:22.911 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:21.844+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2759266635 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7a941046f0 con 0x7f7a84039470 2024-09-10T18:57:22.911 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:22.071+0000 7f7a917fa640 1 -- 172.21.15.119:0/2759266635 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f7a941046f0 con 0x7f7a84039470 2024-09-10T18:57:22.911 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:22.073+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2759266635 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7a84039470 msgr2=0x7f7a8403b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:22.911 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:22.073+0000 7f7a9ac42640 1 --2- 172.21.15.119:0/2759266635 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7a84039470 0x7f7a8403b920 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7a80006fd0 tx=0x7f7a80006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:22.911 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:22.073+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2759266635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a94075480 msgr2=0x7f7a9410f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:22.911 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:22.073+0000 7f7a9ac42640 1 --2- 172.21.15.119:0/2759266635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a94075480 0x7f7a9410f2b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7a88000c00 tx=0x7f7a88008040 comp rx=0 tx=0).stop 2024-09-10T18:57:22.911 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:22.073+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2759266635 shutdown_connections 2024-09-10T18:57:22.911 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:22.073+0000 7f7a9ac42640 1 --2- 172.21.15.119:0/2759266635 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7a84039470 0x7f7a8403b920 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:22.911 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:22.073+0000 7f7a9ac42640 1 --2- 172.21.15.119:0/2759266635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a94075480 0x7f7a9410f2b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:22.912 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:22.073+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2759266635 >> 172.21.15.119:0/2759266635 conn(0x7f7a940facc0 msgr2=0x7f7a940fb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:22.912 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:22.073+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2759266635 shutdown_connections 2024-09-10T18:57:22.912 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:22.073+0000 7f7a9ac42640 1 -- 172.21.15.119:0/2759266635 wait complete. 2024-09-10T18:57:22.912 INFO:teuthology.orchestra.run.smithi119.stdout:Fetching dashboard port number... 2024-09-10T18:57:23.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:23 smithi119 ceph-mon[29577]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:57:23.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:23 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:24.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/345834293' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-10T18:57:24.683 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stdout 8443 2024-09-10T18:57:24.684 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.765+0000 7f880b280640 1 Processor -- start 2024-09-10T18:57:24.684 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.765+0000 7f880b280640 1 -- start start 2024-09-10T18:57:24.684 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.765+0000 7f880b280640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88041006c0 0x7f8804100ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:24.684 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.765+0000 7f880b280640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8804101010 con 0x7f88041006c0 2024-09-10T18:57:24.684 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.765+0000 7f880901c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88041006c0 0x7f8804100ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:24.684 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.765+0000 7f880901c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88041006c0 0x7f8804100ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42804/0 (socket says 172.21.15.119:42804) 2024-09-10T18:57:24.684 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.765+0000 7f880901c640 1 -- 172.21.15.119:0/1418621115 learned_addr learned my addr 172.21.15.119:0/1418621115 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:24.684 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.765+0000 7f880901c640 1 -- 172.21.15.119:0/1418621115 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8804101150 con 0x7f88041006c0 2024-09-10T18:57:24.684 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.766+0000 7f880901c640 1 --2- 172.21.15.119:0/1418621115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88041006c0 0x7f8804100ad0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8800009840 tx=0x7f88000091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5420b651103f9c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:24.684 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.766+0000 7f87fbfff640 1 -- 172.21.15.119:0/1418621115 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f880002ae60 con 0x7f88041006c0 2024-09-10T18:57:24.685 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.766+0000 7f87fbfff640 1 -- 172.21.15.119:0/1418621115 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f88000328b0 con 0x7f88041006c0 2024-09-10T18:57:24.685 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.766+0000 7f87fbfff640 1 -- 172.21.15.119:0/1418621115 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8800031740 con 0x7f88041006c0 2024-09-10T18:57:24.685 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.767+0000 7f880b280640 1 -- 172.21.15.119:0/1418621115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88041006c0 msgr2=0x7f8804100ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:24.685 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.767+0000 7f880b280640 1 --2- 172.21.15.119:0/1418621115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88041006c0 0x7f8804100ad0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8800009840 tx=0x7f88000091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:24.685 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.767+0000 7f880b280640 1 -- 172.21.15.119:0/1418621115 shutdown_connections 2024-09-10T18:57:24.685 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.767+0000 7f880b280640 1 --2- 172.21.15.119:0/1418621115 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88041006c0 0x7f8804100ad0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:24.685 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.767+0000 7f880b280640 1 -- 172.21.15.119:0/1418621115 >> 172.21.15.119:0/1418621115 conn(0x7f88040fbea0 msgr2=0x7f88040fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:24.685 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.767+0000 7f880b280640 1 -- 172.21.15.119:0/1418621115 shutdown_connections 2024-09-10T18:57:24.685 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.767+0000 7f880b280640 1 -- 172.21.15.119:0/1418621115 wait complete. 2024-09-10T18:57:24.686 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.767+0000 7f880b280640 1 Processor -- start 2024-09-10T18:57:24.686 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.767+0000 7f880b280640 1 -- start start 2024-09-10T18:57:24.686 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.768+0000 7f880b280640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88041006c0 0x7f88040774b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:24.686 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.768+0000 7f880b280640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88040779f0 con 0x7f88041006c0 2024-09-10T18:57:24.686 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.768+0000 7f880901c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88041006c0 0x7f88040774b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:24.686 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.768+0000 7f880901c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88041006c0 0x7f88040774b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42812/0 (socket says 172.21.15.119:42812) 2024-09-10T18:57:24.686 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.768+0000 7f880901c640 1 -- 172.21.15.119:0/345834293 learned_addr learned my addr 172.21.15.119:0/345834293 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:24.686 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.768+0000 7f880901c640 1 -- 172.21.15.119:0/345834293 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8800009450 con 0x7f88041006c0 2024-09-10T18:57:24.686 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.768+0000 7f880901c640 1 --2- 172.21.15.119:0/345834293 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88041006c0 0x7f88040774b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8800032670 tx=0x7f88000091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:24.686 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.768+0000 7f87f9ffb640 1 -- 172.21.15.119:0/345834293 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8800031c40 con 0x7f88041006c0 2024-09-10T18:57:24.686 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.768+0000 7f87f9ffb640 1 -- 172.21.15.119:0/345834293 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f8800031da0 con 0x7f88041006c0 2024-09-10T18:57:24.686 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.768+0000 7f880b280640 1 -- 172.21.15.119:0/345834293 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8804075a80 con 0x7f88041006c0 2024-09-10T18:57:24.686 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.768+0000 7f87f9ffb640 1 -- 172.21.15.119:0/345834293 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f880003bea0 con 0x7f88041006c0 2024-09-10T18:57:24.687 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.768+0000 7f880b280640 1 -- 172.21.15.119:0/345834293 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8804075ef0 con 0x7f88041006c0 2024-09-10T18:57:24.687 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.769+0000 7f87f9ffb640 1 -- 172.21.15.119:0/345834293 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 12) v1 ==== 45470+0+0 (secure 0 0 0) 0x7f8800039040 con 0x7f88041006c0 2024-09-10T18:57:24.687 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.769+0000 7f87f9ffb640 1 --2- 172.21.15.119:0/345834293 >> v2:172.21.15.119:6800/1094129222 conn(0x7f87f0039470 0x7f87f003b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:24.687 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.769+0000 7f87f9ffb640 1 -- 172.21.15.119:0/345834293 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f880006c860 con 0x7f88041006c0 2024-09-10T18:57:24.687 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.769+0000 7f880b280640 1 -- 172.21.15.119:0/345834293 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f880404b720 con 0x7f88041006c0 2024-09-10T18:57:24.687 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.770+0000 7f880881b640 1 --2- 172.21.15.119:0/345834293 >> v2:172.21.15.119:6800/1094129222 conn(0x7f87f0039470 0x7f87f003b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:24.687 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.770+0000 7f880881b640 1 --2- 172.21.15.119:0/345834293 >> v2:172.21.15.119:6800/1094129222 conn(0x7f87f0039470 0x7f87f003b920 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f87f4006fd0 tx=0x7f87f4006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:24.687 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.774+0000 7f87f9ffb640 1 -- 172.21.15.119:0/345834293 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f880003f080 con 0x7f88041006c0 2024-09-10T18:57:24.687 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.929+0000 7f880b280640 1 -- 172.21.15.119:0/345834293 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f88041024f0 con 0x7f88041006c0 2024-09-10T18:57:24.687 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.929+0000 7f87f9ffb640 1 -- 172.21.15.119:0/345834293 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f8800037020 con 0x7f88041006c0 2024-09-10T18:57:24.687 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.932+0000 7f880b280640 1 -- 172.21.15.119:0/345834293 >> v2:172.21.15.119:6800/1094129222 conn(0x7f87f0039470 msgr2=0x7f87f003b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:24.687 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.932+0000 7f880b280640 1 --2- 172.21.15.119:0/345834293 >> v2:172.21.15.119:6800/1094129222 conn(0x7f87f0039470 0x7f87f003b920 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f87f4006fd0 tx=0x7f87f4006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:24.687 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.932+0000 7f880b280640 1 -- 172.21.15.119:0/345834293 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88041006c0 msgr2=0x7f88040774b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:24.688 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.932+0000 7f880b280640 1 --2- 172.21.15.119:0/345834293 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88041006c0 0x7f88040774b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8800032670 tx=0x7f88000091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:24.688 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.932+0000 7f880b280640 1 -- 172.21.15.119:0/345834293 shutdown_connections 2024-09-10T18:57:24.688 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.932+0000 7f880b280640 1 --2- 172.21.15.119:0/345834293 >> v2:172.21.15.119:6800/1094129222 conn(0x7f87f0039470 0x7f87f003b920 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:24.688 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.932+0000 7f880b280640 1 --2- 172.21.15.119:0/345834293 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88041006c0 0x7f88040774b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:24.688 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.932+0000 7f880b280640 1 -- 172.21.15.119:0/345834293 >> 172.21.15.119:0/345834293 conn(0x7f88040fbea0 msgr2=0x7f88040fc980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:24.688 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.932+0000 7f880b280640 1 -- 172.21.15.119:0/345834293 shutdown_connections 2024-09-10T18:57:24.688 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:23.932+0000 7f880b280640 1 -- 172.21.15.119:0/345834293 wait complete. 2024-09-10T18:57:24.691 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T18:57:24.691 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout disabled 2024-09-10T18:57:24.699 INFO:teuthology.orchestra.run.smithi119.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T18:57:24.699 INFO:teuthology.orchestra.run.smithi119.stdout:systemctl: stdout inactive 2024-09-10T18:57:24.699 INFO:teuthology.orchestra.run.smithi119.stdout:firewalld.service is not enabled 2024-09-10T18:57:24.700 INFO:teuthology.orchestra.run.smithi119.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-10T18:57:24.702 INFO:teuthology.orchestra.run.smithi119.stdout:Ceph Dashboard is now available at: 2024-09-10T18:57:24.702 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:57:24.702 INFO:teuthology.orchestra.run.smithi119.stdout: URL: https://smithi119.front.sepia.ceph.com:8443/ 2024-09-10T18:57:24.702 INFO:teuthology.orchestra.run.smithi119.stdout: User: admin 2024-09-10T18:57:24.702 INFO:teuthology.orchestra.run.smithi119.stdout: Password: i1bgstwcju 2024-09-10T18:57:24.702 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:57:24.702 INFO:teuthology.orchestra.run.smithi119.stdout:Saving cluster configuration to /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config directory 2024-09-10T18:57:24.703 INFO:teuthology.orchestra.run.smithi119.stdout:Enabling autotune for osd_memory_target 2024-09-10T18:57:25.843 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.132+0000 7f2fd9295640 1 Processor -- start 2024-09-10T18:57:25.848 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.132+0000 7f2fd9295640 1 -- start start 2024-09-10T18:57:25.848 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.132+0000 7f2fd9295640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41005b0 0x7f2fd41009c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:25.848 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.132+0000 7f2fd9295640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd4100f00 con 0x7f2fd41005b0 2024-09-10T18:57:25.848 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.132+0000 7f2fd2ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41005b0 0x7f2fd41009c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:25.848 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.132+0000 7f2fd2ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41005b0 0x7f2fd41009c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42826/0 (socket says 172.21.15.119:42826) 2024-09-10T18:57:25.848 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.132+0000 7f2fd2ffd640 1 -- 172.21.15.119:0/2716589786 learned_addr learned my addr 172.21.15.119:0/2716589786 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:25.848 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.133+0000 7f2fd2ffd640 1 -- 172.21.15.119:0/2716589786 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd4101040 con 0x7f2fd41005b0 2024-09-10T18:57:25.848 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.133+0000 7f2fd2ffd640 1 --2- 172.21.15.119:0/2716589786 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41005b0 0x7f2fd41009c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009e50 tx=0x7f2fc4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6af70685d3bfe90d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:25.848 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.134+0000 7f2fd1ffb640 1 -- 172.21.15.119:0/2716589786 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fc4031400 con 0x7f2fd41005b0 2024-09-10T18:57:25.849 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.134+0000 7f2fd1ffb640 1 -- 172.21.15.119:0/2716589786 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f2fc402d050 con 0x7f2fd41005b0 2024-09-10T18:57:25.849 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.134+0000 7f2fd1ffb640 1 -- 172.21.15.119:0/2716589786 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fc4007690 con 0x7f2fd41005b0 2024-09-10T18:57:25.849 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.134+0000 7f2fd9295640 1 -- 172.21.15.119:0/2716589786 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41005b0 msgr2=0x7f2fd41009c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:25.849 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.134+0000 7f2fd9295640 1 --2- 172.21.15.119:0/2716589786 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41005b0 0x7f2fd41009c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009e50 tx=0x7f2fc4009840 comp rx=0 tx=0).stop 2024-09-10T18:57:25.849 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.134+0000 7f2fd9295640 1 -- 172.21.15.119:0/2716589786 shutdown_connections 2024-09-10T18:57:25.849 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.134+0000 7f2fd9295640 1 --2- 172.21.15.119:0/2716589786 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41005b0 0x7f2fd41009c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:25.849 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.134+0000 7f2fd9295640 1 -- 172.21.15.119:0/2716589786 >> 172.21.15.119:0/2716589786 conn(0x7f2fd40fbd30 msgr2=0x7f2fd40fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:25.849 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.134+0000 7f2fd9295640 1 -- 172.21.15.119:0/2716589786 shutdown_connections 2024-09-10T18:57:25.849 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.134+0000 7f2fd9295640 1 -- 172.21.15.119:0/2716589786 wait complete. 2024-09-10T18:57:25.849 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.135+0000 7f2fd9295640 1 Processor -- start 2024-09-10T18:57:25.849 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.135+0000 7f2fd9295640 1 -- start start 2024-09-10T18:57:25.849 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.135+0000 7f2fd9295640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41005b0 0x7f2fd410f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:25.849 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.135+0000 7f2fd9295640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd410f7b0 con 0x7f2fd41005b0 2024-09-10T18:57:25.849 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.135+0000 7f2fd2ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41005b0 0x7f2fd410f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.135+0000 7f2fd2ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41005b0 0x7f2fd410f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42838/0 (socket says 172.21.15.119:42838) 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.135+0000 7f2fd2ffd640 1 -- 172.21.15.119:0/2376998800 learned_addr learned my addr 172.21.15.119:0/2376998800 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.135+0000 7f2fd2ffd640 1 -- 172.21.15.119:0/2376998800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc4009ae0 con 0x7f2fd41005b0 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.135+0000 7f2fd2ffd640 1 --2- 172.21.15.119:0/2376998800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41005b0 0x7f2fd410f270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4007030 tx=0x7f2fc4031f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.136+0000 7f2fcb7fe640 1 -- 172.21.15.119:0/2376998800 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fc4008930 con 0x7f2fd41005b0 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.136+0000 7f2fcb7fe640 1 -- 172.21.15.119:0/2376998800 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f2fc4008f00 con 0x7f2fd41005b0 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.136+0000 7f2fcb7fe640 1 -- 172.21.15.119:0/2376998800 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2fc403bb70 con 0x7f2fd41005b0 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.136+0000 7f2fd9295640 1 -- 172.21.15.119:0/2376998800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd410f9b0 con 0x7f2fd41005b0 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.136+0000 7f2fd9295640 1 -- 172.21.15.119:0/2376998800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd410fe20 con 0x7f2fd41005b0 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.137+0000 7f2fcb7fe640 1 -- 172.21.15.119:0/2376998800 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 45514+0+0 (secure 0 0 0) 0x7f2fc4038050 con 0x7f2fd41005b0 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.137+0000 7f2fd9295640 1 -- 172.21.15.119:0/2376998800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd404b720 con 0x7f2fd41005b0 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.137+0000 7f2fcb7fe640 1 --2- 172.21.15.119:0/2376998800 >> v2:172.21.15.119:6800/1094129222 conn(0x7f2fc0039560 0x7f2fc003ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.137+0000 7f2fcb7fe640 1 -- 172.21.15.119:0/2376998800 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2fc406c970 con 0x7f2fd41005b0 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.137+0000 7f2fd27fc640 1 --2- 172.21.15.119:0/2376998800 >> v2:172.21.15.119:6800/1094129222 conn(0x7f2fc0039560 0x7f2fc003ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:25.850 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.137+0000 7f2fd27fc640 1 --2- 172.21.15.119:0/2376998800 >> v2:172.21.15.119:6800/1094129222 conn(0x7f2fc0039560 0x7f2fc003ba10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc006fd0 tx=0x7f2fbc006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:25.851 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.141+0000 7f2fcb7fe640 1 -- 172.21.15.119:0/2376998800 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f2fc4008a90 con 0x7f2fd41005b0 2024-09-10T18:57:25.851 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.297+0000 7f2fd9295640 1 -- 172.21.15.119:0/2376998800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f2fd4102370 con 0x7f2fd41005b0 2024-09-10T18:57:25.851 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.298+0000 7f2fcb7fe640 1 -- 172.21.15.119:0/2376998800 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f2fc4041030 con 0x7f2fd41005b0 2024-09-10T18:57:25.851 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.299+0000 7f2fd9295640 1 -- 172.21.15.119:0/2376998800 >> v2:172.21.15.119:6800/1094129222 conn(0x7f2fc0039560 msgr2=0x7f2fc003ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:25.851 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.299+0000 7f2fd9295640 1 --2- 172.21.15.119:0/2376998800 >> v2:172.21.15.119:6800/1094129222 conn(0x7f2fc0039560 0x7f2fc003ba10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc006fd0 tx=0x7f2fbc006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:25.851 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.299+0000 7f2fd9295640 1 -- 172.21.15.119:0/2376998800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41005b0 msgr2=0x7f2fd410f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:25.851 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.299+0000 7f2fd9295640 1 --2- 172.21.15.119:0/2376998800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41005b0 0x7f2fd410f270 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4007030 tx=0x7f2fc4031f90 comp rx=0 tx=0).stop 2024-09-10T18:57:25.851 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.299+0000 7f2fd9295640 1 -- 172.21.15.119:0/2376998800 shutdown_connections 2024-09-10T18:57:25.851 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.299+0000 7f2fd9295640 1 --2- 172.21.15.119:0/2376998800 >> v2:172.21.15.119:6800/1094129222 conn(0x7f2fc0039560 0x7f2fc003ba10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:25.851 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.299+0000 7f2fd9295640 1 --2- 172.21.15.119:0/2376998800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fd41005b0 0x7f2fd410f270 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:25.851 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.299+0000 7f2fd9295640 1 -- 172.21.15.119:0/2376998800 >> 172.21.15.119:0/2376998800 conn(0x7f2fd40fbd30 msgr2=0x7f2fd40fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:25.851 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.299+0000 7f2fd9295640 1 -- 172.21.15.119:0/2376998800 shutdown_connections 2024-09-10T18:57:25.851 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:25.299+0000 7f2fd9295640 1 -- 172.21.15.119:0/2376998800 wait complete. 2024-09-10T18:57:26.267 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:26 smithi119 ceph-mon[29577]: mgrmap e13: y(active, since 6s) 2024-09-10T18:57:27.105 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.266+0000 7ff1f5ffb640 1 Processor -- start 2024-09-10T18:57:27.106 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.266+0000 7ff1f5ffb640 1 -- start start 2024-09-10T18:57:27.106 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.266+0000 7ff1f5ffb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1f0074be0 0x7ff1f01031f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:27.106 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.266+0000 7ff1f5ffb640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f0103730 con 0x7ff1f0074be0 2024-09-10T18:57:27.106 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.266+0000 7ff1ef7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1f0074be0 0x7ff1f01031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:27.106 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.266+0000 7ff1ef7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1f0074be0 0x7ff1f01031f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42854/0 (socket says 172.21.15.119:42854) 2024-09-10T18:57:27.106 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.266+0000 7ff1ef7fe640 1 -- 172.21.15.119:0/1497994573 learned_addr learned my addr 172.21.15.119:0/1497994573 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:27.106 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.266+0000 7ff1ef7fe640 1 -- 172.21.15.119:0/1497994573 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f0103870 con 0x7ff1f0074be0 2024-09-10T18:57:27.106 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.267+0000 7ff1ef7fe640 1 --2- 172.21.15.119:0/1497994573 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1f0074be0 0x7ff1f01031f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009840 tx=0x7ff1e00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb18f18e58e8092 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:27.106 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.267+0000 7ff1ee7fc640 1 -- 172.21.15.119:0/1497994573 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1e002add0 con 0x7ff1f0074be0 2024-09-10T18:57:27.106 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.267+0000 7ff1ee7fc640 1 -- 172.21.15.119:0/1497994573 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7ff1e00328b0 con 0x7ff1f0074be0 2024-09-10T18:57:27.106 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.267+0000 7ff1ee7fc640 1 -- 172.21.15.119:0/1497994573 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1e0031690 con 0x7ff1f0074be0 2024-09-10T18:57:27.106 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.268+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/1497994573 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1f0074be0 msgr2=0x7ff1f01031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:27.107 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.268+0000 7ff1f5ffb640 1 --2- 172.21.15.119:0/1497994573 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1f0074be0 0x7ff1f01031f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009840 tx=0x7ff1e00091d0 comp rx=0 tx=0).stop 2024-09-10T18:57:27.107 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.268+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/1497994573 shutdown_connections 2024-09-10T18:57:27.107 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.268+0000 7ff1f5ffb640 1 --2- 172.21.15.119:0/1497994573 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1f0074be0 0x7ff1f01031f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:27.107 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.268+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/1497994573 >> 172.21.15.119:0/1497994573 conn(0x7ff1f00fabc0 msgr2=0x7ff1f00fd030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:27.107 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.268+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/1497994573 shutdown_connections 2024-09-10T18:57:27.107 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.268+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/1497994573 wait complete. 2024-09-10T18:57:27.107 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.268+0000 7ff1f5ffb640 1 Processor -- start 2024-09-10T18:57:27.107 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.269+0000 7ff1f5ffb640 1 -- start start 2024-09-10T18:57:27.107 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.269+0000 7ff1f5ffb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1f0074be0 0x7ff1f0100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:27.107 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.269+0000 7ff1f5ffb640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f0101060 con 0x7ff1f0074be0 2024-09-10T18:57:27.107 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.269+0000 7ff1ef7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1f0074be0 0x7ff1f0100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:27.107 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.269+0000 7ff1ef7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1f0074be0 0x7ff1f0100b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42858/0 (socket says 172.21.15.119:42858) 2024-09-10T18:57:27.107 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.269+0000 7ff1ef7fe640 1 -- 172.21.15.119:0/429603277 learned_addr learned my addr 172.21.15.119:0/429603277 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:27.107 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.269+0000 7ff1ef7fe640 1 -- 172.21.15.119:0/429603277 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e0009450 con 0x7ff1f0074be0 2024-09-10T18:57:27.108 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.269+0000 7ff1ef7fe640 1 --2- 172.21.15.119:0/429603277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1f0074be0 0x7ff1f0100b20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0000c00 tx=0x7ff1e0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:27.108 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.269+0000 7ff1e7fff640 1 -- 172.21.15.119:0/429603277 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1e0031c50 con 0x7ff1f0074be0 2024-09-10T18:57:27.108 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.269+0000 7ff1e7fff640 1 -- 172.21.15.119:0/429603277 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7ff1e0031db0 con 0x7ff1f0074be0 2024-09-10T18:57:27.108 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.269+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/429603277 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f00ff0f0 con 0x7ff1f0074be0 2024-09-10T18:57:27.108 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.269+0000 7ff1e7fff640 1 -- 172.21.15.119:0/429603277 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff1e003a3a0 con 0x7ff1f0074be0 2024-09-10T18:57:27.108 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.269+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/429603277 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f00ff560 con 0x7ff1f0074be0 2024-09-10T18:57:27.108 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.270+0000 7ff1e7fff640 1 -- 172.21.15.119:0/429603277 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 45514+0+0 (secure 0 0 0) 0x7ff1e0039070 con 0x7ff1f0074be0 2024-09-10T18:57:27.108 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.270+0000 7ff1e7fff640 1 --2- 172.21.15.119:0/429603277 >> v2:172.21.15.119:6800/1094129222 conn(0x7ff1dc039560 0x7ff1dc03ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:27.108 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.270+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/429603277 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f005ed70 con 0x7ff1f0074be0 2024-09-10T18:57:27.108 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.271+0000 7ff1e7fff640 1 -- 172.21.15.119:0/429603277 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff1e006cb50 con 0x7ff1f0074be0 2024-09-10T18:57:27.108 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.271+0000 7ff1eeffd640 1 --2- 172.21.15.119:0/429603277 >> v2:172.21.15.119:6800/1094129222 conn(0x7ff1dc039560 0x7ff1dc03ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:27.109 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.271+0000 7ff1eeffd640 1 --2- 172.21.15.119:0/429603277 >> v2:172.21.15.119:6800/1094129222 conn(0x7ff1dc039560 0x7ff1dc03ba10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8006fd0 tx=0x7ff1d8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:27.109 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.275+0000 7ff1e7fff640 1 -- 172.21.15.119:0/429603277 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7ff1e0044400 con 0x7ff1f0074be0 2024-09-10T18:57:27.109 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.512+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/429603277 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7ff1f004b720 con 0x7ff1f0074be0 2024-09-10T18:57:27.109 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.514+0000 7ff1e7fff640 1 -- 172.21.15.119:0/429603277 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7ff1e003f020 con 0x7ff1f0074be0 2024-09-10T18:57:27.109 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-10T18:57:27.109 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.516+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/429603277 >> v2:172.21.15.119:6800/1094129222 conn(0x7ff1dc039560 msgr2=0x7ff1dc03ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:27.109 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.516+0000 7ff1f5ffb640 1 --2- 172.21.15.119:0/429603277 >> v2:172.21.15.119:6800/1094129222 conn(0x7ff1dc039560 0x7ff1dc03ba10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff1d8006fd0 tx=0x7ff1d8006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:27.109 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.516+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/429603277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1f0074be0 msgr2=0x7ff1f0100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:27.109 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.516+0000 7ff1f5ffb640 1 --2- 172.21.15.119:0/429603277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1f0074be0 0x7ff1f0100b20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0000c00 tx=0x7ff1e0008040 comp rx=0 tx=0).stop 2024-09-10T18:57:27.109 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.516+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/429603277 shutdown_connections 2024-09-10T18:57:27.109 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.517+0000 7ff1f5ffb640 1 --2- 172.21.15.119:0/429603277 >> v2:172.21.15.119:6800/1094129222 conn(0x7ff1dc039560 0x7ff1dc03ba10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:27.110 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.517+0000 7ff1f5ffb640 1 --2- 172.21.15.119:0/429603277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1f0074be0 0x7ff1f0100b20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:27.110 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.517+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/429603277 >> 172.21.15.119:0/429603277 conn(0x7ff1f00fabc0 msgr2=0x7ff1f00fb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:27.110 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.517+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/429603277 shutdown_connections 2024-09-10T18:57:27.110 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/bin/ceph: stderr 2024-09-10T18:57:26.517+0000 7ff1f5ffb640 1 -- 172.21.15.119:0/429603277 wait complete. 2024-09-10T18:57:27.110 INFO:teuthology.orchestra.run.smithi119.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-10T18:57:27.110 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:57:27.110 INFO:teuthology.orchestra.run.smithi119.stdout: sudo /sbin/cephadm shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-10T18:57:27.110 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:57:27.110 INFO:teuthology.orchestra.run.smithi119.stdout:Or, if you are only running a single cluster on this host: 2024-09-10T18:57:27.110 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:57:27.110 INFO:teuthology.orchestra.run.smithi119.stdout: sudo /sbin/cephadm shell 2024-09-10T18:57:27.110 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:57:27.111 INFO:teuthology.orchestra.run.smithi119.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-10T18:57:27.111 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:57:27.111 INFO:teuthology.orchestra.run.smithi119.stdout: ceph telemetry on 2024-09-10T18:57:27.111 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:57:27.111 INFO:teuthology.orchestra.run.smithi119.stdout:For more information see: 2024-09-10T18:57:27.111 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:57:27.111 INFO:teuthology.orchestra.run.smithi119.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-10T18:57:27.111 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:57:27.111 INFO:teuthology.orchestra.run.smithi119.stdout:Bootstrap complete. 2024-09-10T18:57:27.142 INFO:tasks.cephadm:Fetching config... 2024-09-10T18:57:27.142 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T18:57:27.142 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-10T18:57:27.160 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-10T18:57:27.160 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T18:57:27.160 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-10T18:57:27.216 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-10T18:57:27.216 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T18:57:27.216 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-10T18:57:27.282 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-10T18:57:27.282 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T18:57:27.282 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-10T18:57:27.338 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-10T18:57:27.338 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2N27W4TumTIc76bjdOh9eW79+HQvmxaekWa83O2GPrBJax8C3Nf5Halpxkx/hAUF4oiY3TDJYWoNcfz+x9U6o4RX7VBl0/W7Puv2OLsbK2NDwwLtpqA7SReYCxd2hwoBkiptVPp7N6twHr5uMyStlPvWlQjyO+WiTsOauYzA5nTvX/mDecNQ3920klrymbLtNGbBJiXACX/okQ42JpqmIuvjm/SSGFx7yZcSREQK5/MdoFbv3vWT3p80UoFAtJMXqyQ+Lw1uvW0j7jbff2k4tcBK35UYjhmBCAJ5atEFuQc+SEVAkBi4z6/M1u3W8xEL1alzUFzYIGCfoQIPiTqeHbn5rUeEEH8LJnNKQDURgSlNB2lZU4SIii/N9Io3Z/AlsBO8KA4NZ9JcgRk7wzb71SEylcWVkwQW8F4jg81gLL8qCN0S4b9anBZ/KQVgcbYXRZjb2VWcXS36aHk5kHPesgtDikJbMnM1aCjv9blBnPRXEcHrht38bIctk6qUPeCs= ceph-3afe9804-6fa6-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-10T18:57:27.421 INFO:teuthology.orchestra.run.smithi119.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2N27W4TumTIc76bjdOh9eW79+HQvmxaekWa83O2GPrBJax8C3Nf5Halpxkx/hAUF4oiY3TDJYWoNcfz+x9U6o4RX7VBl0/W7Puv2OLsbK2NDwwLtpqA7SReYCxd2hwoBkiptVPp7N6twHr5uMyStlPvWlQjyO+WiTsOauYzA5nTvX/mDecNQ3920klrymbLtNGbBJiXACX/okQ42JpqmIuvjm/SSGFx7yZcSREQK5/MdoFbv3vWT3p80UoFAtJMXqyQ+Lw1uvW0j7jbff2k4tcBK35UYjhmBCAJ5atEFuQc+SEVAkBi4z6/M1u3W8xEL1alzUFzYIGCfoQIPiTqeHbn5rUeEEH8LJnNKQDURgSlNB2lZU4SIii/N9Io3Z/AlsBO8KA4NZ9JcgRk7wzb71SEylcWVkwQW8F4jg81gLL8qCN0S4b9anBZ/KQVgcbYXRZjb2VWcXS36aHk5kHPesgtDikJbMnM1aCjv9blBnPRXEcHrht38bIctk6qUPeCs= ceph-3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:57:27.432 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2N27W4TumTIc76bjdOh9eW79+HQvmxaekWa83O2GPrBJax8C3Nf5Halpxkx/hAUF4oiY3TDJYWoNcfz+x9U6o4RX7VBl0/W7Puv2OLsbK2NDwwLtpqA7SReYCxd2hwoBkiptVPp7N6twHr5uMyStlPvWlQjyO+WiTsOauYzA5nTvX/mDecNQ3920klrymbLtNGbBJiXACX/okQ42JpqmIuvjm/SSGFx7yZcSREQK5/MdoFbv3vWT3p80UoFAtJMXqyQ+Lw1uvW0j7jbff2k4tcBK35UYjhmBCAJ5atEFuQc+SEVAkBi4z6/M1u3W8xEL1alzUFzYIGCfoQIPiTqeHbn5rUeEEH8LJnNKQDURgSlNB2lZU4SIii/N9Io3Z/AlsBO8KA4NZ9JcgRk7wzb71SEylcWVkwQW8F4jg81gLL8qCN0S4b9anBZ/KQVgcbYXRZjb2VWcXS36aHk5kHPesgtDikJbMnM1aCjv9blBnPRXEcHrht38bIctk6qUPeCs= ceph-3afe9804-6fa6-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-10T18:57:27.489 INFO:teuthology.orchestra.run.smithi194.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC2N27W4TumTIc76bjdOh9eW79+HQvmxaekWa83O2GPrBJax8C3Nf5Halpxkx/hAUF4oiY3TDJYWoNcfz+x9U6o4RX7VBl0/W7Puv2OLsbK2NDwwLtpqA7SReYCxd2hwoBkiptVPp7N6twHr5uMyStlPvWlQjyO+WiTsOauYzA5nTvX/mDecNQ3920klrymbLtNGbBJiXACX/okQ42JpqmIuvjm/SSGFx7yZcSREQK5/MdoFbv3vWT3p80UoFAtJMXqyQ+Lw1uvW0j7jbff2k4tcBK35UYjhmBCAJ5atEFuQc+SEVAkBi4z6/M1u3W8xEL1alzUFzYIGCfoQIPiTqeHbn5rUeEEH8LJnNKQDURgSlNB2lZU4SIii/N9Io3Z/AlsBO8KA4NZ9JcgRk7wzb71SEylcWVkwQW8F4jg81gLL8qCN0S4b9anBZ/KQVgcbYXRZjb2VWcXS36aHk5kHPesgtDikJbMnM1aCjv9blBnPRXEcHrht38bIctk6qUPeCs= ceph-3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:57:27.502 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-10T18:57:27.744 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:27.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/429603277' entity='client.admin' 2024-09-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.096+0000 7f7176e3a640 1 -- 172.21.15.119:0/312352245 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7170065650 msgr2=0x7f71700feff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.096+0000 7f7176e3a640 1 --2- 172.21.15.119:0/312352245 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7170065650 0x7f71700feff0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f716c004850 tx=0x7f716c009da0 comp rx=0 tx=0).stop 2024-09-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.096+0000 7f7176e3a640 1 -- 172.21.15.119:0/312352245 shutdown_connections 2024-09-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.096+0000 7f7176e3a640 1 --2- 172.21.15.119:0/312352245 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7170065650 0x7f71700feff0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.096+0000 7f7176e3a640 1 -- 172.21.15.119:0/312352245 >> 172.21.15.119:0/312352245 conn(0x7f71700f6ac0 msgr2=0x7f71700f8ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.096+0000 7f7176e3a640 1 -- 172.21.15.119:0/312352245 shutdown_connections 2024-09-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.096+0000 7f7176e3a640 1 -- 172.21.15.119:0/312352245 wait complete. 2024-09-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.097+0000 7f7176e3a640 1 Processor -- start 2024-09-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.097+0000 7f7176e3a640 1 -- start start 2024-09-10T18:57:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.097+0000 7f7176e3a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7170065650 0x7f717010bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.097+0000 7f7176e3a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f717010c440 con 0x7f7170065650 2024-09-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.097+0000 7f7174bd6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7170065650 0x7f717010bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.098+0000 7f7174bd6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7170065650 0x7f717010bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42876/0 (socket says 172.21.15.119:42876) 2024-09-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.098+0000 7f7174bd6640 1 -- 172.21.15.119:0/4125571766 learned_addr learned my addr 172.21.15.119:0/4125571766 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.098+0000 7f7174bd6640 1 -- 172.21.15.119:0/4125571766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716c005040 con 0x7f7170065650 2024-09-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.098+0000 7f7174bd6640 1 --2- 172.21.15.119:0/4125571766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7170065650 0x7f717010bf00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f716c02a6d0 tx=0x7f716c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.098+0000 7f71657fa640 1 -- 172.21.15.119:0/4125571766 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f716c038070 con 0x7f7170065650 2024-09-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.098+0000 7f71657fa640 1 -- 172.21.15.119:0/4125571766 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f716c0098a0 con 0x7f7170065650 2024-09-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.098+0000 7f7176e3a640 1 -- 172.21.15.119:0/4125571766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f717010c640 con 0x7f7170065650 2024-09-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.098+0000 7f71657fa640 1 -- 172.21.15.119:0/4125571766 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f716c03c5b0 con 0x7f7170065650 2024-09-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.098+0000 7f7176e3a640 1 -- 172.21.15.119:0/4125571766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f717010d090 con 0x7f7170065650 2024-09-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.099+0000 7f71657fa640 1 -- 172.21.15.119:0/4125571766 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 45514+0+0 (secure 0 0 0) 0x7f716c030020 con 0x7f7170065650 2024-09-10T18:57:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.099+0000 7f7176e3a640 1 -- 172.21.15.119:0/4125571766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7170101db0 con 0x7f7170065650 2024-09-10T18:57:30.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.099+0000 7f71657fa640 1 --2- 172.21.15.119:0/4125571766 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7158039560 0x7f715803ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:30.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.099+0000 7f71657fa640 1 -- 172.21.15.119:0/4125571766 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f716c06c310 con 0x7f7170065650 2024-09-10T18:57:30.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.099+0000 7f7167fff640 1 --2- 172.21.15.119:0/4125571766 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7158039560 0x7f715803ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:30.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.100+0000 7f7167fff640 1 --2- 172.21.15.119:0/4125571766 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7158039560 0x7f715803ba10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7160006fd0 tx=0x7f7160006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:30.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.104+0000 7f71657fa640 1 -- 172.21.15.119:0/4125571766 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f716c03c710 con 0x7f7170065650 2024-09-10T18:57:30.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.262+0000 7f7176e3a640 1 -- 172.21.15.119:0/4125571766 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f7170101600 con 0x7f7170065650 2024-09-10T18:57:30.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.268+0000 7f71657fa640 1 -- 172.21.15.119:0/4125571766 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f716c044390 con 0x7f7170065650 2024-09-10T18:57:30.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.270+0000 7f7176e3a640 1 -- 172.21.15.119:0/4125571766 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7158039560 msgr2=0x7f715803ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:30.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.270+0000 7f7176e3a640 1 --2- 172.21.15.119:0/4125571766 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7158039560 0x7f715803ba10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7160006fd0 tx=0x7f7160006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:30.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.270+0000 7f7176e3a640 1 -- 172.21.15.119:0/4125571766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7170065650 msgr2=0x7f717010bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:30.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.270+0000 7f7176e3a640 1 --2- 172.21.15.119:0/4125571766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7170065650 0x7f717010bf00 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f716c02a6d0 tx=0x7f716c009da0 comp rx=0 tx=0).stop 2024-09-10T18:57:30.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.270+0000 7f7176e3a640 1 -- 172.21.15.119:0/4125571766 shutdown_connections 2024-09-10T18:57:30.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.270+0000 7f7176e3a640 1 --2- 172.21.15.119:0/4125571766 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7158039560 0x7f715803ba10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:30.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.270+0000 7f7176e3a640 1 --2- 172.21.15.119:0/4125571766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7170065650 0x7f717010bf00 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:30.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.270+0000 7f7176e3a640 1 -- 172.21.15.119:0/4125571766 >> 172.21.15.119:0/4125571766 conn(0x7f71700f6ac0 msgr2=0x7f71700f7350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:30.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.270+0000 7f7176e3a640 1 -- 172.21.15.119:0/4125571766 shutdown_connections 2024-09-10T18:57:30.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:30.270+0000 7f7176e3a640 1 -- 172.21.15.119:0/4125571766 wait complete. 2024-09-10T18:57:30.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:30 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:30.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:30 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:30.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:30 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi119", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:57:30.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:30 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:30.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:30 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:57:30.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:30 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:30.939 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-10T18:57:30.939 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-10T18:57:31.457 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4125571766' entity='client.admin' 2024-09-10T18:57:31.457 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:57:31.457 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:32.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.982+0000 7f22c64c8640 1 -- 172.21.15.119:0/3727405722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22c00fb0d0 msgr2=0x7f22c00fb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:32.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.982+0000 7f22c64c8640 1 --2- 172.21.15.119:0/3727405722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22c00fb0d0 0x7f22c00fb4e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f22b0004850 tx=0x7f22b0009da0 comp rx=0 tx=0).stop 2024-09-10T18:57:32.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.982+0000 7f22c64c8640 1 -- 172.21.15.119:0/3727405722 shutdown_connections 2024-09-10T18:57:32.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.982+0000 7f22c64c8640 1 --2- 172.21.15.119:0/3727405722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22c00fb0d0 0x7f22c00fb4e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:32.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.982+0000 7f22c64c8640 1 -- 172.21.15.119:0/3727405722 >> 172.21.15.119:0/3727405722 conn(0x7f22c00f6890 msgr2=0x7f22c00f8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:32.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.982+0000 7f22c64c8640 1 -- 172.21.15.119:0/3727405722 shutdown_connections 2024-09-10T18:57:32.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.982+0000 7f22c64c8640 1 -- 172.21.15.119:0/3727405722 wait complete. 2024-09-10T18:57:32.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.983+0000 7f22c64c8640 1 Processor -- start 2024-09-10T18:57:32.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.983+0000 7f22c64c8640 1 -- start start 2024-09-10T18:57:32.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.983+0000 7f22c64c8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22c00fb0d0 0x7f22c010ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:32.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.983+0000 7f22c64c8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c010b060 con 0x7f22c00fb0d0 2024-09-10T18:57:32.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.983+0000 7f22bffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22c00fb0d0 0x7f22c010ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:32.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.983+0000 7f22bffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22c00fb0d0 0x7f22c010ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42948/0 (socket says 172.21.15.119:42948) 2024-09-10T18:57:32.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.983+0000 7f22bffff640 1 -- 172.21.15.119:0/3223371167 learned_addr learned my addr 172.21.15.119:0/3223371167 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:32.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.983+0000 7f22bffff640 1 -- 172.21.15.119:0/3223371167 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b0005040 con 0x7f22c00fb0d0 2024-09-10T18:57:32.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.983+0000 7f22bffff640 1 --2- 172.21.15.119:0/3223371167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22c00fb0d0 0x7f22c010ab20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f22b0004820 tx=0x7f22b0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:32.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.984+0000 7f22bd7fa640 1 -- 172.21.15.119:0/3223371167 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22b0038070 con 0x7f22c00fb0d0 2024-09-10T18:57:32.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.984+0000 7f22bd7fa640 1 -- 172.21.15.119:0/3223371167 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f22b00098a0 con 0x7f22c00fb0d0 2024-09-10T18:57:32.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.984+0000 7f22c64c8640 1 -- 172.21.15.119:0/3223371167 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c010b260 con 0x7f22c00fb0d0 2024-09-10T18:57:32.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.984+0000 7f22bd7fa640 1 -- 172.21.15.119:0/3223371167 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f22b003c5b0 con 0x7f22c00fb0d0 2024-09-10T18:57:32.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.984+0000 7f22c64c8640 1 -- 172.21.15.119:0/3223371167 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c010b670 con 0x7f22c00fb0d0 2024-09-10T18:57:32.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.985+0000 7f22bd7fa640 1 -- 172.21.15.119:0/3223371167 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 45514+0+0 (secure 0 0 0) 0x7f22b0030020 con 0x7f22c00fb0d0 2024-09-10T18:57:32.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.985+0000 7f22c64c8640 1 -- 172.21.15.119:0/3223371167 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c005ed70 con 0x7f22c00fb0d0 2024-09-10T18:57:32.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.985+0000 7f22bd7fa640 1 --2- 172.21.15.119:0/3223371167 >> v2:172.21.15.119:6800/1094129222 conn(0x7f22ac039560 0x7f22ac03ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:32.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.985+0000 7f22bd7fa640 1 -- 172.21.15.119:0/3223371167 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f22b006bd50 con 0x7f22c00fb0d0 2024-09-10T18:57:32.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.985+0000 7f22b7fff640 1 --2- 172.21.15.119:0/3223371167 >> v2:172.21.15.119:6800/1094129222 conn(0x7f22ac039560 0x7f22ac03ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:32.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.985+0000 7f22b7fff640 1 --2- 172.21.15.119:0/3223371167 >> v2:172.21.15.119:6800/1094129222 conn(0x7f22ac039560 0x7f22ac03ba10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f22a8006fd0 tx=0x7f22a8006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:32.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:32.989+0000 7f22bd7fa640 1 -- 172.21.15.119:0/3223371167 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f22b003c710 con 0x7f22c00fb0d0 2024-09-10T18:57:33.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:33.148+0000 7f22c64c8640 1 -- 172.21.15.119:0/3223371167 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f22c010be50 con 0x7f22ac039560 2024-09-10T18:57:33.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:33.151+0000 7f22bd7fa640 1 -- 172.21.15.119:0/3223371167 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f22c010be50 con 0x7f22ac039560 2024-09-10T18:57:33.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:33.152+0000 7f22c64c8640 1 -- 172.21.15.119:0/3223371167 >> v2:172.21.15.119:6800/1094129222 conn(0x7f22ac039560 msgr2=0x7f22ac03ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:33.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:33.152+0000 7f22c64c8640 1 --2- 172.21.15.119:0/3223371167 >> v2:172.21.15.119:6800/1094129222 conn(0x7f22ac039560 0x7f22ac03ba10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f22a8006fd0 tx=0x7f22a8006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:33.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:33.152+0000 7f22c64c8640 1 -- 172.21.15.119:0/3223371167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22c00fb0d0 msgr2=0x7f22c010ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:33.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:33.152+0000 7f22c64c8640 1 --2- 172.21.15.119:0/3223371167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22c00fb0d0 0x7f22c010ab20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f22b0004820 tx=0x7f22b0009da0 comp rx=0 tx=0).stop 2024-09-10T18:57:33.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:33.153+0000 7f22c64c8640 1 -- 172.21.15.119:0/3223371167 shutdown_connections 2024-09-10T18:57:33.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:33.153+0000 7f22c64c8640 1 --2- 172.21.15.119:0/3223371167 >> v2:172.21.15.119:6800/1094129222 conn(0x7f22ac039560 0x7f22ac03ba10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:33.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:33.153+0000 7f22c64c8640 1 --2- 172.21.15.119:0/3223371167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22c00fb0d0 0x7f22c010ab20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:33.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:33.153+0000 7f22c64c8640 1 -- 172.21.15.119:0/3223371167 >> 172.21.15.119:0/3223371167 conn(0x7f22c00f6890 msgr2=0x7f22c00f7540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:33.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:33.153+0000 7f22c64c8640 1 -- 172.21.15.119:0/3223371167 shutdown_connections 2024-09-10T18:57:33.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:33.153+0000 7f22c64c8640 1 -- 172.21.15.119:0/3223371167 wait complete. 2024-09-10T18:57:33.965 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi194 2024-09-10T18:57:33.965 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-10T18:57:33.965 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.conf 2024-09-10T18:57:33.984 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-10T18:57:33.984 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:57:34.042 INFO:tasks.cephadm:Adding host smithi194 to orchestrator... 2024-09-10T18:57:34.042 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch host add smithi194 2024-09-10T18:57:34.278 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:34 smithi119 ceph-mon[29577]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:57:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:57:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:57:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:57:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:34 smithi119 ceph-mon[29577]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-10T18:57:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:34 smithi119 ceph-mon[29577]: Updating smithi119:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:57:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:34 smithi119 ceph-mon[29577]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:57:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:35.388 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:35 smithi119 ceph-mon[29577]: Updating smithi119:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:57:35.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.842+0000 7fcde6eb9640 1 -- 172.21.15.119:0/692577421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcde01010e0 msgr2=0x7fcde01014b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:35.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.842+0000 7fcde6eb9640 1 --2- 172.21.15.119:0/692577421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcde01010e0 0x7fcde01014b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4004850 tx=0x7fcdd4009da0 comp rx=0 tx=0).stop 2024-09-10T18:57:35.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.842+0000 7fcde6eb9640 1 -- 172.21.15.119:0/692577421 shutdown_connections 2024-09-10T18:57:35.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.842+0000 7fcde6eb9640 1 --2- 172.21.15.119:0/692577421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcde01010e0 0x7fcde01014b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:35.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.842+0000 7fcde6eb9640 1 -- 172.21.15.119:0/692577421 >> 172.21.15.119:0/692577421 conn(0x7fcde00710b0 msgr2=0x7fcde00714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:35.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.842+0000 7fcde6eb9640 1 -- 172.21.15.119:0/692577421 shutdown_connections 2024-09-10T18:57:35.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.843+0000 7fcde6eb9640 1 -- 172.21.15.119:0/692577421 wait complete. 2024-09-10T18:57:35.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.843+0000 7fcde6eb9640 1 Processor -- start 2024-09-10T18:57:35.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.843+0000 7fcde6eb9640 1 -- start start 2024-09-10T18:57:35.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.843+0000 7fcde6eb9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcde01010e0 0x7fcde0111c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:35.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.843+0000 7fcde6eb9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde01121a0 con 0x7fcde01010e0 2024-09-10T18:57:35.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.843+0000 7fcde4c55640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcde01010e0 0x7fcde0111c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:35.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.844+0000 7fcde4c55640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcde01010e0 0x7fcde0111c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42962/0 (socket says 172.21.15.119:42962) 2024-09-10T18:57:35.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.844+0000 7fcde4c55640 1 -- 172.21.15.119:0/3401839261 learned_addr learned my addr 172.21.15.119:0/3401839261 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:35.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.844+0000 7fcde4c55640 1 -- 172.21.15.119:0/3401839261 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd4005040 con 0x7fcde01010e0 2024-09-10T18:57:35.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.844+0000 7fcde4c55640 1 --2- 172.21.15.119:0/3401839261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcde01010e0 0x7fcde0111c60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4003080 tx=0x7fcdd4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:35.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.844+0000 7fcddd7fa640 1 -- 172.21.15.119:0/3401839261 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdd4038070 con 0x7fcde01010e0 2024-09-10T18:57:35.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.844+0000 7fcddd7fa640 1 -- 172.21.15.119:0/3401839261 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fcdd40093b0 con 0x7fcde01010e0 2024-09-10T18:57:35.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.844+0000 7fcde6eb9640 1 -- 172.21.15.119:0/3401839261 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde01123a0 con 0x7fcde01010e0 2024-09-10T18:57:35.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.844+0000 7fcddd7fa640 1 -- 172.21.15.119:0/3401839261 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdd403c410 con 0x7fcde01010e0 2024-09-10T18:57:35.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.844+0000 7fcde6eb9640 1 -- 172.21.15.119:0/3401839261 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde0112810 con 0x7fcde01010e0 2024-09-10T18:57:35.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.845+0000 7fcddd7fa640 1 -- 172.21.15.119:0/3401839261 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 45514+0+0 (secure 0 0 0) 0x7fcdd4030020 con 0x7fcde01010e0 2024-09-10T18:57:35.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.845+0000 7fcde6eb9640 1 -- 172.21.15.119:0/3401839261 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde00fc850 con 0x7fcde01010e0 2024-09-10T18:57:35.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.845+0000 7fcddd7fa640 1 --2- 172.21.15.119:0/3401839261 >> v2:172.21.15.119:6800/1094129222 conn(0x7fcdd0039510 0x7fcdd003b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:35.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.845+0000 7fcddd7fa640 1 -- 172.21.15.119:0/3401839261 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcdd406bc80 con 0x7fcde01010e0 2024-09-10T18:57:35.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.845+0000 7fcddffff640 1 --2- 172.21.15.119:0/3401839261 >> v2:172.21.15.119:6800/1094129222 conn(0x7fcdd0039510 0x7fcdd003b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:35.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.846+0000 7fcddffff640 1 --2- 172.21.15.119:0/3401839261 >> v2:172.21.15.119:6800/1094129222 conn(0x7fcdd0039510 0x7fcdd003b9c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc006fd0 tx=0x7fcdcc006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:35.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:35.849+0000 7fcddd7fa640 1 -- 172.21.15.119:0/3401839261 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fcdd403a6d0 con 0x7fcde01010e0 2024-09-10T18:57:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:36.006+0000 7fcde6eb9640 1 -- 172.21.15.119:0/3401839261 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}) v1 -- 0x7fcde010d370 con 0x7fcdd0039510 2024-09-10T18:57:37.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:36 smithi119 ceph-mon[29577]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:57:38.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:37 smithi119 ceph-mon[29577]: Deploying cephadm binary to smithi194 2024-09-10T18:57:38.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:38.677+0000 7fcddd7fa640 1 -- 172.21.15.119:0/3401839261 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fcde010d370 con 0x7fcdd0039510 2024-09-10T18:57:38.679 INFO:teuthology.orchestra.run.smithi119.stdout:Added host 'smithi194' with addr '172.21.15.194' 2024-09-10T18:57:38.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:38.679+0000 7fcde6eb9640 1 -- 172.21.15.119:0/3401839261 >> v2:172.21.15.119:6800/1094129222 conn(0x7fcdd0039510 msgr2=0x7fcdd003b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:38.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:38.679+0000 7fcde6eb9640 1 --2- 172.21.15.119:0/3401839261 >> v2:172.21.15.119:6800/1094129222 conn(0x7fcdd0039510 0x7fcdd003b9c0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc006fd0 tx=0x7fcdcc006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:38.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:38.679+0000 7fcde6eb9640 1 -- 172.21.15.119:0/3401839261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcde01010e0 msgr2=0x7fcde0111c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:38.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:38.679+0000 7fcde6eb9640 1 --2- 172.21.15.119:0/3401839261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcde01010e0 0x7fcde0111c60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4003080 tx=0x7fcdd4009da0 comp rx=0 tx=0).stop 2024-09-10T18:57:38.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:38.679+0000 7fcde6eb9640 1 -- 172.21.15.119:0/3401839261 shutdown_connections 2024-09-10T18:57:38.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:38.679+0000 7fcde6eb9640 1 --2- 172.21.15.119:0/3401839261 >> v2:172.21.15.119:6800/1094129222 conn(0x7fcdd0039510 0x7fcdd003b9c0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:38.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:38.679+0000 7fcde6eb9640 1 --2- 172.21.15.119:0/3401839261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcde01010e0 0x7fcde0111c60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:38.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:38.679+0000 7fcde6eb9640 1 -- 172.21.15.119:0/3401839261 >> 172.21.15.119:0/3401839261 conn(0x7fcde00710b0 msgr2=0x7fcde0064b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:38.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:38.680+0000 7fcde6eb9640 1 -- 172.21.15.119:0/3401839261 shutdown_connections 2024-09-10T18:57:38.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:38.680+0000 7fcde6eb9640 1 -- 172.21.15.119:0/3401839261 wait complete. 2024-09-10T18:57:39.284 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch host ls --format=json 2024-09-10T18:57:39.518 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:39.938 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:39 smithi119 ceph-mon[29577]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:39.938 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:39.938 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:39 smithi119 ceph-mon[29577]: Added host smithi194 2024-09-10T18:57:39.938 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:57:39.938 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:39.938 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:41.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.128+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/2227997808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102200 msgr2=0x7fc9ec1025d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:41.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.128+0000 7fc9f3ba0640 1 --2- 172.21.15.119:0/2227997808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102200 0x7fc9ec1025d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8004850 tx=0x7fc9e8009da0 comp rx=0 tx=0).stop 2024-09-10T18:57:41.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.128+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/2227997808 shutdown_connections 2024-09-10T18:57:41.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.128+0000 7fc9f3ba0640 1 --2- 172.21.15.119:0/2227997808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102200 0x7fc9ec1025d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:41.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.128+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/2227997808 >> 172.21.15.119:0/2227997808 conn(0x7fc9ec0f7dd0 msgr2=0x7fc9ec0fa1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:41.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.128+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/2227997808 shutdown_connections 2024-09-10T18:57:41.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.128+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/2227997808 wait complete. 2024-09-10T18:57:41.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.128+0000 7fc9f3ba0640 1 Processor -- start 2024-09-10T18:57:41.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.128+0000 7fc9f3ba0640 1 -- start start 2024-09-10T18:57:41.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.129+0000 7fc9f3ba0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102200 0x7fc9ec111c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:41.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.129+0000 7fc9f3ba0640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec1121c0 con 0x7fc9ec102200 2024-09-10T18:57:41.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.129+0000 7fc9f193c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102200 0x7fc9ec111c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:41.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.129+0000 7fc9f193c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102200 0x7fc9ec111c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42988/0 (socket says 172.21.15.119:42988) 2024-09-10T18:57:41.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.129+0000 7fc9f193c640 1 -- 172.21.15.119:0/541185717 learned_addr learned my addr 172.21.15.119:0/541185717 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:41.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.129+0000 7fc9f193c640 1 -- 172.21.15.119:0/541185717 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e8005040 con 0x7fc9ec102200 2024-09-10T18:57:41.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.129+0000 7fc9f193c640 1 --2- 172.21.15.119:0/541185717 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102200 0x7fc9ec111c80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8004820 tx=0x7fc9e8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:41.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.129+0000 7fc9e27fc640 1 -- 172.21.15.119:0/541185717 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9e8038070 con 0x7fc9ec102200 2024-09-10T18:57:41.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.129+0000 7fc9e27fc640 1 -- 172.21.15.119:0/541185717 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fc9e80092e0 con 0x7fc9ec102200 2024-09-10T18:57:41.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.129+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/541185717 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ec1123c0 con 0x7fc9ec102200 2024-09-10T18:57:41.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.129+0000 7fc9e27fc640 1 -- 172.21.15.119:0/541185717 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc9e803c410 con 0x7fc9ec102200 2024-09-10T18:57:41.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.129+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/541185717 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ec1127d0 con 0x7fc9ec102200 2024-09-10T18:57:41.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.130+0000 7fc9e27fc640 1 -- 172.21.15.119:0/541185717 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 45514+0+0 (secure 0 0 0) 0x7fc9e8030020 con 0x7fc9ec102200 2024-09-10T18:57:41.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.130+0000 7fc9e27fc640 1 --2- 172.21.15.119:0/541185717 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc9d8039510 0x7fc9d803b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:41.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.130+0000 7fc9e27fc640 1 -- 172.21.15.119:0/541185717 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc9e806bbf0 con 0x7fc9ec102200 2024-09-10T18:57:41.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.130+0000 7fc9f113b640 1 --2- 172.21.15.119:0/541185717 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc9d8039510 0x7fc9d803b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:41.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.130+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/541185717 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9ec0fd970 con 0x7fc9ec102200 2024-09-10T18:57:41.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.131+0000 7fc9f113b640 1 --2- 172.21.15.119:0/541185717 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc9d8039510 0x7fc9d803b9c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc006fd0 tx=0x7fc9dc006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:41.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.135+0000 7fc9e27fc640 1 -- 172.21.15.119:0/541185717 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fc9e803ad30 con 0x7fc9ec102200 2024-09-10T18:57:41.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.297+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/541185717 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc9ec1059f0 con 0x7fc9d8039510 2024-09-10T18:57:41.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.298+0000 7fc9e27fc640 1 -- 172.21.15.119:0/541185717 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7fc9ec1059f0 con 0x7fc9d8039510 2024-09-10T18:57:41.299 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T18:57:41.299 INFO:teuthology.orchestra.run.smithi119.stdout:[{"addr": "172.21.15.119", "hostname": "smithi119", "labels": [], "status": ""}, {"addr": "172.21.15.194", "hostname": "smithi194", "labels": [], "status": ""}] 2024-09-10T18:57:41.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.300+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/541185717 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc9d8039510 msgr2=0x7fc9d803b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:41.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.300+0000 7fc9f3ba0640 1 --2- 172.21.15.119:0/541185717 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc9d8039510 0x7fc9d803b9c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc9dc006fd0 tx=0x7fc9dc006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:41.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.300+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/541185717 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102200 msgr2=0x7fc9ec111c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:41.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.300+0000 7fc9f3ba0640 1 --2- 172.21.15.119:0/541185717 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102200 0x7fc9ec111c80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc9e8004820 tx=0x7fc9e8009da0 comp rx=0 tx=0).stop 2024-09-10T18:57:41.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.300+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/541185717 shutdown_connections 2024-09-10T18:57:41.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:41 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:41.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:41 smithi119 ceph-mon[29577]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:41.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.300+0000 7fc9f3ba0640 1 --2- 172.21.15.119:0/541185717 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc9d8039510 0x7fc9d803b9c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:41.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.300+0000 7fc9f3ba0640 1 --2- 172.21.15.119:0/541185717 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9ec102200 0x7fc9ec111c80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:41.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.300+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/541185717 >> 172.21.15.119:0/541185717 conn(0x7fc9ec0f7dd0 msgr2=0x7fc9ec0f8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:41.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.300+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/541185717 shutdown_connections 2024-09-10T18:57:41.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:41.300+0000 7fc9f3ba0640 1 -- 172.21.15.119:0/541185717 wait complete. 2024-09-10T18:57:41.890 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-10T18:57:41.891 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd crush tunables default 2024-09-10T18:57:42.125 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:57:42.246 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:42 smithi119 ceph-mon[29577]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T18:57:43.189 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:43 smithi119 ceph-mon[29577]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:43.189 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:43.189 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:43.189 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:43.189 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:43.189 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:57:43.189 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:57:43.190 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:57:43.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.714+0000 7f5c78e22640 1 -- 172.21.15.119:0/1855189712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c741021f0 msgr2=0x7f5c741025c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:43.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.714+0000 7f5c78e22640 1 --2- 172.21.15.119:0/1855189712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c741021f0 0x7f5c741025c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5c64004850 tx=0x7f5c64009da0 comp rx=0 tx=0).stop 2024-09-10T18:57:43.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.714+0000 7f5c78e22640 1 -- 172.21.15.119:0/1855189712 shutdown_connections 2024-09-10T18:57:43.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.714+0000 7f5c78e22640 1 --2- 172.21.15.119:0/1855189712 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c741021f0 0x7f5c741025c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:43.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.714+0000 7f5c78e22640 1 -- 172.21.15.119:0/1855189712 >> 172.21.15.119:0/1855189712 conn(0x7f5c740717c0 msgr2=0x7f5c74071bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:43.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.714+0000 7f5c78e22640 1 -- 172.21.15.119:0/1855189712 shutdown_connections 2024-09-10T18:57:43.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.714+0000 7f5c78e22640 1 -- 172.21.15.119:0/1855189712 wait complete. 2024-09-10T18:57:43.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.715+0000 7f5c78e22640 1 Processor -- start 2024-09-10T18:57:43.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.715+0000 7f5c78e22640 1 -- start start 2024-09-10T18:57:43.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.715+0000 7f5c78e22640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c741021f0 0x7f5c740fba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:43.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.715+0000 7f5c78e22640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c740fbf70 con 0x7f5c741021f0 2024-09-10T18:57:43.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.715+0000 7f5c7259c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c741021f0 0x7f5c740fba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.715+0000 7f5c7259c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c741021f0 0x7f5c740fba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45712/0 (socket says 172.21.15.119:45712) 2024-09-10T18:57:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.715+0000 7f5c7259c640 1 -- 172.21.15.119:0/2446877516 learned_addr learned my addr 172.21.15.119:0/2446877516 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:57:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.715+0000 7f5c7259c640 1 -- 172.21.15.119:0/2446877516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c64005040 con 0x7f5c741021f0 2024-09-10T18:57:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.715+0000 7f5c7259c640 1 --2- 172.21.15.119:0/2446877516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c741021f0 0x7f5c740fba30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5c64004820 tx=0x7f5c64009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.715+0000 7f5c62ffd640 1 -- 172.21.15.119:0/2446877516 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c64038070 con 0x7f5c741021f0 2024-09-10T18:57:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.715+0000 7f5c62ffd640 1 -- 172.21.15.119:0/2446877516 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f5c640092e0 con 0x7f5c741021f0 2024-09-10T18:57:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.715+0000 7f5c78e22640 1 -- 172.21.15.119:0/2446877516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c740fc170 con 0x7f5c741021f0 2024-09-10T18:57:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.716+0000 7f5c62ffd640 1 -- 172.21.15.119:0/2446877516 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5c6403c410 con 0x7f5c741021f0 2024-09-10T18:57:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.716+0000 7f5c78e22640 1 -- 172.21.15.119:0/2446877516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c740fc580 con 0x7f5c741021f0 2024-09-10T18:57:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.716+0000 7f5c62ffd640 1 -- 172.21.15.119:0/2446877516 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 45514+0+0 (secure 0 0 0) 0x7f5c64030020 con 0x7f5c741021f0 2024-09-10T18:57:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.716+0000 7f5c78e22640 1 -- 172.21.15.119:0/2446877516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c740f8e60 con 0x7f5c741021f0 2024-09-10T18:57:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.717+0000 7f5c62ffd640 1 --2- 172.21.15.119:0/2446877516 >> v2:172.21.15.119:6800/1094129222 conn(0x7f5c5c039560 0x7f5c5c03ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.717+0000 7f5c62ffd640 1 -- 172.21.15.119:0/2446877516 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5c6406baf0 con 0x7f5c741021f0 2024-09-10T18:57:43.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.717+0000 7f5c71d9b640 1 --2- 172.21.15.119:0/2446877516 >> v2:172.21.15.119:6800/1094129222 conn(0x7f5c5c039560 0x7f5c5c03ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:43.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.717+0000 7f5c71d9b640 1 --2- 172.21.15.119:0/2446877516 >> v2:172.21.15.119:6800/1094129222 conn(0x7f5c5c039560 0x7f5c5c03ba10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5c68006fd0 tx=0x7f5c68006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:43.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.721+0000 7f5c62ffd640 1 -- 172.21.15.119:0/2446877516 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f5c6403c570 con 0x7f5c741021f0 2024-09-10T18:57:43.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:43.883+0000 7f5c78e22640 1 -- 172.21.15.119:0/2446877516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5c740652f0 con 0x7f5c741021f0 2024-09-10T18:57:44.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:44.661+0000 7f5c62ffd640 1 -- 172.21.15.119:0/2446877516 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5c6403bb60 con 0x7f5c741021f0 2024-09-10T18:57:44.663 INFO:teuthology.orchestra.run.smithi119.stderr:adjusted tunables profile to default 2024-09-10T18:57:44.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:44.664+0000 7f5c78e22640 1 -- 172.21.15.119:0/2446877516 >> v2:172.21.15.119:6800/1094129222 conn(0x7f5c5c039560 msgr2=0x7f5c5c03ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:44.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:44.664+0000 7f5c78e22640 1 --2- 172.21.15.119:0/2446877516 >> v2:172.21.15.119:6800/1094129222 conn(0x7f5c5c039560 0x7f5c5c03ba10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5c68006fd0 tx=0x7f5c68006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:44.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:44.664+0000 7f5c78e22640 1 -- 172.21.15.119:0/2446877516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c741021f0 msgr2=0x7f5c740fba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:44.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:44.664+0000 7f5c78e22640 1 --2- 172.21.15.119:0/2446877516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c741021f0 0x7f5c740fba30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5c64004820 tx=0x7f5c64009da0 comp rx=0 tx=0).stop 2024-09-10T18:57:44.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:44.664+0000 7f5c78e22640 1 -- 172.21.15.119:0/2446877516 shutdown_connections 2024-09-10T18:57:44.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:44.664+0000 7f5c78e22640 1 --2- 172.21.15.119:0/2446877516 >> v2:172.21.15.119:6800/1094129222 conn(0x7f5c5c039560 0x7f5c5c03ba10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:44.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:44.664+0000 7f5c78e22640 1 --2- 172.21.15.119:0/2446877516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c741021f0 0x7f5c740fba30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:44.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:44.664+0000 7f5c78e22640 1 -- 172.21.15.119:0/2446877516 >> 172.21.15.119:0/2446877516 conn(0x7f5c740717c0 msgr2=0x7f5c740f9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:44.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:44.664+0000 7f5c78e22640 1 -- 172.21.15.119:0/2446877516 shutdown_connections 2024-09-10T18:57:44.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:57:44.664+0000 7f5c78e22640 1 -- 172.21.15.119:0/2446877516 wait complete. 2024-09-10T18:57:44.979 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:44 smithi119 ceph-mon[29577]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-10T18:57:44.979 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:44 smithi119 ceph-mon[29577]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:57:44.979 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:44 smithi119 ceph-mon[29577]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:57:44.979 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:44 smithi119 ceph-mon[29577]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:57:44.979 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:44 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:44.979 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:44 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:44.979 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:44 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:44.979 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2446877516' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-10T18:57:45.252 INFO:tasks.cephadm:Adding mon.a on smithi119 2024-09-10T18:57:45.252 INFO:tasks.cephadm:Adding mon.c on smithi119 2024-09-10T18:57:45.252 INFO:tasks.cephadm:Adding mon.b on smithi194 2024-09-10T18:57:45.252 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch apply mon '3;smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi194:172.21.15.194=b' 2024-09-10T18:57:45.494 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:57:46.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:45 smithi119 ceph-mon[29577]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:46.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2446877516' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-10T18:57:46.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:45 smithi119 ceph-mon[29577]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:57:47.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.104+0000 7fbe795ba640 1 -- 172.21.15.194:0/3197380366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe74102400 msgr2=0x7fbe741027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:47.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.104+0000 7fbe795ba640 1 --2- 172.21.15.194:0/3197380366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe74102400 0x7fbe741027d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbe64004850 tx=0x7fbe64009da0 comp rx=0 tx=0).stop 2024-09-10T18:57:47.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.104+0000 7fbe795ba640 1 -- 172.21.15.194:0/3197380366 shutdown_connections 2024-09-10T18:57:47.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.104+0000 7fbe795ba640 1 --2- 172.21.15.194:0/3197380366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe74102400 0x7fbe741027d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:47.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.104+0000 7fbe795ba640 1 -- 172.21.15.194:0/3197380366 >> 172.21.15.194:0/3197380366 conn(0x7fbe74071790 msgr2=0x7fbe74071b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:47.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.104+0000 7fbe795ba640 1 -- 172.21.15.194:0/3197380366 shutdown_connections 2024-09-10T18:57:47.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.104+0000 7fbe795ba640 1 -- 172.21.15.194:0/3197380366 wait complete. 2024-09-10T18:57:47.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.104+0000 7fbe795ba640 1 Processor -- start 2024-09-10T18:57:47.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.105+0000 7fbe795ba640 1 -- start start 2024-09-10T18:57:47.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.105+0000 7fbe795ba640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe74102400 0x7fbe74111e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:47.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.105+0000 7fbe795ba640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe741123c0 con 0x7fbe74102400 2024-09-10T18:57:47.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.105+0000 7fbe72ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe74102400 0x7fbe74111e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:47.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.105+0000 7fbe72ffd640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe74102400 0x7fbe74111e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.194:54798/0 (socket says 172.21.15.194:54798) 2024-09-10T18:57:47.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.105+0000 7fbe72ffd640 1 -- 172.21.15.194:0/162027743 learned_addr learned my addr 172.21.15.194:0/162027743 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T18:57:47.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.105+0000 7fbe72ffd640 1 -- 172.21.15.194:0/162027743 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe64005040 con 0x7fbe74102400 2024-09-10T18:57:47.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.105+0000 7fbe72ffd640 1 --2- 172.21.15.194:0/162027743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe74102400 0x7fbe74111e80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbe64003080 tx=0x7fbe64009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:47.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.106+0000 7fbe6b7fe640 1 -- 172.21.15.194:0/162027743 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe64038070 con 0x7fbe74102400 2024-09-10T18:57:47.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.106+0000 7fbe795ba640 1 -- 172.21.15.194:0/162027743 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe741125c0 con 0x7fbe74102400 2024-09-10T18:57:47.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.106+0000 7fbe6b7fe640 1 -- 172.21.15.194:0/162027743 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fbe640093b0 con 0x7fbe74102400 2024-09-10T18:57:47.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.106+0000 7fbe6b7fe640 1 -- 172.21.15.194:0/162027743 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fbe6403c410 con 0x7fbe74102400 2024-09-10T18:57:47.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.106+0000 7fbe795ba640 1 -- 172.21.15.194:0/162027743 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe74112a30 con 0x7fbe74102400 2024-09-10T18:57:47.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.107+0000 7fbe6b7fe640 1 -- 172.21.15.194:0/162027743 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 45514+0+0 (secure 0 0 0) 0x7fbe64030020 con 0x7fbe74102400 2024-09-10T18:57:47.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.107+0000 7fbe795ba640 1 -- 172.21.15.194:0/162027743 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe740fdb70 con 0x7fbe74102400 2024-09-10T18:57:47.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.107+0000 7fbe6b7fe640 1 --2- 172.21.15.194:0/162027743 >> v2:172.21.15.119:6800/1094129222 conn(0x7fbe60039510 0x7fbe6003b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:47.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.107+0000 7fbe6b7fe640 1 -- 172.21.15.194:0/162027743 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbe6406bd10 con 0x7fbe74102400 2024-09-10T18:57:47.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.107+0000 7fbe727fc640 1 --2- 172.21.15.194:0/162027743 >> v2:172.21.15.119:6800/1094129222 conn(0x7fbe60039510 0x7fbe6003b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:47.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.108+0000 7fbe727fc640 1 --2- 172.21.15.194:0/162027743 >> v2:172.21.15.119:6800/1094129222 conn(0x7fbe60039510 0x7fbe6003b9c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c006fd0 tx=0x7fbe5c006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:47.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.112+0000 7fbe6b7fe640 1 -- 172.21.15.194:0/162027743 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fbe64071020 con 0x7fbe74102400 2024-09-10T18:57:47.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.275+0000 7fbe795ba640 1 -- 172.21.15.194:0/162027743 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi194:172.21.15.194=b", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe74105bf0 con 0x7fbe60039510 2024-09-10T18:57:47.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.281+0000 7fbe6b7fe640 1 -- 172.21.15.194:0/162027743 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fbe74105bf0 con 0x7fbe60039510 2024-09-10T18:57:47.282 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mon update... 2024-09-10T18:57:47.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.282+0000 7fbe795ba640 1 -- 172.21.15.194:0/162027743 >> v2:172.21.15.119:6800/1094129222 conn(0x7fbe60039510 msgr2=0x7fbe6003b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:47.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.282+0000 7fbe795ba640 1 --2- 172.21.15.194:0/162027743 >> v2:172.21.15.119:6800/1094129222 conn(0x7fbe60039510 0x7fbe6003b9c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c006fd0 tx=0x7fbe5c006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:47.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.282+0000 7fbe795ba640 1 -- 172.21.15.194:0/162027743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe74102400 msgr2=0x7fbe74111e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:47.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.282+0000 7fbe795ba640 1 --2- 172.21.15.194:0/162027743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe74102400 0x7fbe74111e80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fbe64003080 tx=0x7fbe64009da0 comp rx=0 tx=0).stop 2024-09-10T18:57:47.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.283+0000 7fbe795ba640 1 -- 172.21.15.194:0/162027743 shutdown_connections 2024-09-10T18:57:47.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.283+0000 7fbe795ba640 1 --2- 172.21.15.194:0/162027743 >> v2:172.21.15.119:6800/1094129222 conn(0x7fbe60039510 0x7fbe6003b9c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:47.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.283+0000 7fbe795ba640 1 --2- 172.21.15.194:0/162027743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe74102400 0x7fbe74111e80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:47.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.283+0000 7fbe795ba640 1 -- 172.21.15.194:0/162027743 >> 172.21.15.194:0/162027743 conn(0x7fbe74071790 msgr2=0x7fbe74103a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:47.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.283+0000 7fbe795ba640 1 -- 172.21.15.194:0/162027743 shutdown_connections 2024-09-10T18:57:47.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:47.283+0000 7fbe795ba640 1 -- 172.21.15.194:0/162027743 wait complete. 2024-09-10T18:57:47.867 DEBUG:teuthology.orchestra.run.smithi119:mon.c> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.c.service 2024-09-10T18:57:47.870 DEBUG:teuthology.orchestra.run.smithi194:mon.b> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.b.service 2024-09-10T18:57:47.872 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-10T18:57:47.872 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T18:57:48.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:47 smithi119 ceph-mon[29577]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:48.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:47 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:48.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:47 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:57:48.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:47 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:57:48.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:47 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:57:48.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:47 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:48.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:47 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:57:48.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:47 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:57:48.497 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:57:49.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:48 smithi119 ceph-mon[29577]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi194:172.21.15.194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:57:49.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:48 smithi119 ceph-mon[29577]: Saving service mon spec with placement smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi194:172.21.15.194=b;count:3 2024-09-10T18:57:49.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:48 smithi119 ceph-mon[29577]: Deploying daemon mon.b on smithi194 2024-09-10T18:57:50.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:49 smithi119 ceph-mon[29577]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:51.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.498+0000 7f6555b92640 1 -- 172.21.15.194:0/220227926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65501010e0 msgr2=0x7f65501014b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:51.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.498+0000 7f6555b92640 1 --2- 172.21.15.194:0/220227926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65501010e0 0x7f65501014b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f6540004850 tx=0x7f6540009da0 comp rx=0 tx=0).stop 2024-09-10T18:57:51.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.498+0000 7f6555b92640 1 -- 172.21.15.194:0/220227926 shutdown_connections 2024-09-10T18:57:51.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.498+0000 7f6555b92640 1 --2- 172.21.15.194:0/220227926 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65501010e0 0x7f65501014b0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:51.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.498+0000 7f6555b92640 1 -- 172.21.15.194:0/220227926 >> 172.21.15.194:0/220227926 conn(0x7f65500710b0 msgr2=0x7f65500714b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:51.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.498+0000 7f6555b92640 1 -- 172.21.15.194:0/220227926 shutdown_connections 2024-09-10T18:57:51.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.498+0000 7f6555b92640 1 -- 172.21.15.194:0/220227926 wait complete. 2024-09-10T18:57:51.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.499+0000 7f6555b92640 1 Processor -- start 2024-09-10T18:57:51.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.499+0000 7f6555b92640 1 -- start start 2024-09-10T18:57:51.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.499+0000 7f6555b92640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65501010e0 0x7f655010cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:51.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.499+0000 7f6555b92640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655010d150 con 0x7f65501010e0 2024-09-10T18:57:51.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.500+0000 7f654f7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65501010e0 0x7f655010cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:51.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.500+0000 7f654f7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65501010e0 0x7f655010cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.194:54814/0 (socket says 172.21.15.194:54814) 2024-09-10T18:57:51.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.500+0000 7f654f7fe640 1 -- 172.21.15.194:0/1010097481 learned_addr learned my addr 172.21.15.194:0/1010097481 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T18:57:51.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.500+0000 7f654f7fe640 1 -- 172.21.15.194:0/1010097481 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6540005040 con 0x7f65501010e0 2024-09-10T18:57:51.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.500+0000 7f654f7fe640 1 --2- 172.21.15.194:0/1010097481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65501010e0 0x7f655010cc10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6540003080 tx=0x7f6540009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:51.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.500+0000 7f6537fff640 1 -- 172.21.15.194:0/1010097481 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6540038070 con 0x7f65501010e0 2024-09-10T18:57:51.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.500+0000 7f6555b92640 1 -- 172.21.15.194:0/1010097481 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f655010d350 con 0x7f65501010e0 2024-09-10T18:57:51.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.500+0000 7f6537fff640 1 -- 172.21.15.194:0/1010097481 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f65400093b0 con 0x7f65501010e0 2024-09-10T18:57:51.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.500+0000 7f6537fff640 1 -- 172.21.15.194:0/1010097481 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f654003c410 con 0x7f65501010e0 2024-09-10T18:57:51.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.500+0000 7f6555b92640 1 -- 172.21.15.194:0/1010097481 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655010d7c0 con 0x7f65501010e0 2024-09-10T18:57:51.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.501+0000 7f6537fff640 1 -- 172.21.15.194:0/1010097481 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 45514+0+0 (secure 0 0 0) 0x7f6540030020 con 0x7f65501010e0 2024-09-10T18:57:51.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.501+0000 7f6537fff640 1 --2- 172.21.15.194:0/1010097481 >> v2:172.21.15.119:6800/1094129222 conn(0x7f6538039510 0x7f653803b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:51.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.502+0000 7f6537fff640 1 -- 172.21.15.194:0/1010097481 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f654006bd10 con 0x7f65501010e0 2024-09-10T18:57:51.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.502+0000 7f6555b92640 1 -- 172.21.15.194:0/1010097481 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65500f9600 con 0x7f65501010e0 2024-09-10T18:57:51.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.502+0000 7f654effd640 1 --2- 172.21.15.194:0/1010097481 >> v2:172.21.15.119:6800/1094129222 conn(0x7f6538039510 0x7f653803b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:51.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.502+0000 7f654effd640 1 --2- 172.21.15.194:0/1010097481 >> v2:172.21.15.119:6800/1094129222 conn(0x7f6538039510 0x7f653803b9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6544006fd0 tx=0x7f6544006db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:51.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.507+0000 7f6537fff640 1 -- 172.21.15.194:0/1010097481 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f65500f9600 con 0x7f65501010e0 2024-09-10T18:57:51.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.742+0000 7f6555b92640 1 -- 172.21.15.194:0/1010097481 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65500f9600 con 0x7f65501010e0 2024-09-10T18:57:51.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.743+0000 7f6537fff640 1 -- 172.21.15.194:0/1010097481 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+735 (secure 0 0 0) 0x7f65500f9600 con 0x7f65501010e0 2024-09-10T18:57:51.744 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:57:51.744 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"3afe9804-6fa6-11ef-bcea-c7b262605968","modified":"2024-09-10T18:56:23.443244Z","created":"2024-09-10T18:56:23.443244Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T18:57:51.745 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-09-10T18:57:51.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.745+0000 7f6555b92640 1 -- 172.21.15.194:0/1010097481 >> v2:172.21.15.119:6800/1094129222 conn(0x7f6538039510 msgr2=0x7f653803b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:51.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.745+0000 7f6555b92640 1 --2- 172.21.15.194:0/1010097481 >> v2:172.21.15.119:6800/1094129222 conn(0x7f6538039510 0x7f653803b9c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6544006fd0 tx=0x7f6544006db0 comp rx=0 tx=0).stop 2024-09-10T18:57:51.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.745+0000 7f6555b92640 1 -- 172.21.15.194:0/1010097481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65501010e0 msgr2=0x7f655010cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:51.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.745+0000 7f6555b92640 1 --2- 172.21.15.194:0/1010097481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65501010e0 0x7f655010cc10 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6540003080 tx=0x7f6540009da0 comp rx=0 tx=0).stop 2024-09-10T18:57:51.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.745+0000 7f6555b92640 1 -- 172.21.15.194:0/1010097481 shutdown_connections 2024-09-10T18:57:51.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.745+0000 7f6555b92640 1 --2- 172.21.15.194:0/1010097481 >> v2:172.21.15.119:6800/1094129222 conn(0x7f6538039510 0x7f653803b9c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:51.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.745+0000 7f6555b92640 1 --2- 172.21.15.194:0/1010097481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65501010e0 0x7f655010cc10 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:51.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.745+0000 7f6555b92640 1 -- 172.21.15.194:0/1010097481 >> 172.21.15.194:0/1010097481 conn(0x7f65500710b0 msgr2=0x7f65500fa0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:51.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.745+0000 7f6555b92640 1 -- 172.21.15.194:0/1010097481 shutdown_connections 2024-09-10T18:57:51.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:51.745+0000 7f6555b92640 1 -- 172.21.15.194:0/1010097481 wait complete. 2024-09-10T18:57:52.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:51 smithi119 ceph-mon[29577]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:52.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 systemd[1]: Starting Ceph mon.b for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T18:57:52.693 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 podman[32425]: 2024-09-10 18:57:52.47146692 +0000 UTC m=+0.015393087 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:57:52.693 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 podman[32425]: 2024-09-10 18:57:52.574099405 +0000 UTC m=+0.118025606 container create 8762862dea5c36023485aa02040f92a7c6ce35c0d06943310900f8dd840dbbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-b, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-10T18:57:53.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/1010097481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:57:53.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 podman[32425]: 2024-09-10 18:57:52.784543746 +0000 UTC m=+0.328469906 container init 8762862dea5c36023485aa02040f92a7c6ce35c0d06943310900f8dd840dbbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, org.label-schema.vendor=CentOS) 2024-09-10T18:57:53.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 podman[32425]: 2024-09-10 18:57:52.788832111 +0000 UTC m=+0.332758268 container start 8762862dea5c36023485aa02040f92a7c6ce35c0d06943310900f8dd840dbbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-b, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:57:53.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: set uid:gid to 167:167 (ceph:ceph) 2024-09-10T18:57:53.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable), process ceph-mon, pid 7 2024-09-10T18:57:53.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: pidfile_write: ignore empty --pid-file 2024-09-10T18:57:53.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: load: jerasure load: lrc 2024-09-10T18:57:53.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: RocksDB version: 6.15.5 2024-09-10T18:57:53.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-10T18:57:53.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Compile date Sep 9 2024 2024-09-10T18:57:53.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: DB SUMMARY 2024-09-10T18:57:53.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: DB Session ID: 8YRQFNDTVF0GQ03BZMF1 2024-09-10T18:57:53.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: CURRENT file: CURRENT 2024-09-10T18:57:53.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: IDENTITY file: IDENTITY 2024-09-10T18:57:53.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-10T18:57:53.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-09-10T18:57:53.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-09-10T18:57:53.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.error_if_exists: 0 2024-09-10T18:57:53.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.create_if_missing: 0 2024-09-10T18:57:53.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.paranoid_checks: 1 2024-09-10T18:57:53.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-10T18:57:53.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.env: 0x55cc419c2440 2024-09-10T18:57:53.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.fs: Posix File System 2024-09-10T18:57:53.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.info_log: 0x55cc43480ec0 2024-09-10T18:57:53.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_file_opening_threads: 16 2024-09-10T18:57:53.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.statistics: (nil) 2024-09-10T18:57:53.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.use_fsync: 0 2024-09-10T18:57:53.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_log_file_size: 0 2024-09-10T18:57:53.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-10T18:57:53.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-10T18:57:53.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.keep_log_file_num: 1000 2024-09-10T18:57:53.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.recycle_log_file_num: 0 2024-09-10T18:57:53.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.allow_fallocate: 1 2024-09-10T18:57:53.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.allow_mmap_reads: 0 2024-09-10T18:57:53.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.allow_mmap_writes: 0 2024-09-10T18:57:53.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.use_direct_reads: 0 2024-09-10T18:57:53.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-10T18:57:53.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.create_missing_column_families: 0 2024-09-10T18:57:53.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.db_log_dir: 2024-09-10T18:57:53.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-b/store.db 2024-09-10T18:57:53.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-10T18:57:53.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-10T18:57:53.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-10T18:57:53.274 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-10T18:57:53.274 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-10T18:57:53.274 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-10T18:57:53.274 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.advise_random_on_open: 1 2024-09-10T18:57:53.274 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.db_write_buffer_size: 0 2024-09-10T18:57:53.274 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.write_buffer_manager: 0x55cc4353e780 2024-09-10T18:57:53.274 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-10T18:57:53.274 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-10T18:57:53.274 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-10T18:57:53.274 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-10T18:57:53.274 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.rate_limiter: (nil) 2024-09-10T18:57:53.274 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-10T18:57:53.274 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.wal_recovery_mode: 2 2024-09-10T18:57:53.274 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.enable_thread_tracking: 0 2024-09-10T18:57:53.275 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.enable_pipelined_write: 0 2024-09-10T18:57:53.275 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.unordered_write: 0 2024-09-10T18:57:53.275 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-10T18:57:53.275 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-10T18:57:53.275 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-10T18:57:53.275 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-10T18:57:53.275 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.row_cache: None 2024-09-10T18:57:53.275 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.wal_filter: None 2024-09-10T18:57:53.275 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-10T18:57:53.275 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.allow_ingest_behind: 0 2024-09-10T18:57:53.275 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.preserve_deletes: 0 2024-09-10T18:57:53.275 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.two_write_queues: 0 2024-09-10T18:57:53.275 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.manual_wal_flush: 0 2024-09-10T18:57:53.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.atomic_flush: 0 2024-09-10T18:57:53.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-10T18:57:53.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-10T18:57:53.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-10T18:57:53.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.log_readahead_size: 0 2024-09-10T18:57:53.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-10T18:57:53.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.best_efforts_recovery: 0 2024-09-10T18:57:53.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-10T18:57:53.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-10T18:57:53.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.allow_data_in_errors: 0 2024-09-10T18:57:53.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.db_host_id: __hostname__ 2024-09-10T18:57:53.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_background_jobs: 2 2024-09-10T18:57:53.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_background_compactions: -1 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_subcompactions: 1 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_total_wal_size: 0 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_open_files: -1 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.bytes_per_sync: 0 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compaction_readahead_size: 0 2024-09-10T18:57:53.277 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_background_flushes: -1 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Compression algorithms supported: 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: kZSTD supported: 0 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: kXpressCompression supported: 0 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: kBZip2Compression supported: 0 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: kLZ4Compression supported: 1 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: kZlibCompression supported: 1 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: kLZ4HCCompression supported: 1 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: kSnappyCompression supported: 1 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.merge_operator: 2024-09-10T18:57:53.278 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compaction_filter: None 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compaction_filter_factory: None 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.sst_partitioner_factory: None 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55cc43427a38) 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout: cache_index_and_filter_blocks: 1 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout: pin_top_level_index_and_filter: 1 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout: index_type: 0 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout: data_block_index_type: 0 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout: index_shortening: 1 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout: hash_index_allow_collision: 1 2024-09-10T18:57:53.279 INFO:journalctl@ceph.mon.b.smithi194.stdout: checksum: 1 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: no_block_cache: 0 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache: 0x55cc4345d1f0 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache_name: BinnedLRUCache 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache_options: 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: capacity : 536870912 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: num_shard_bits : 4 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: strict_capacity_limit : 0 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: high_pri_pool_ratio: 0.000 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache_compressed: (nil) 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: persistent_cache: (nil) 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_size: 4096 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_size_deviation: 10 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_restart_interval: 16 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: index_block_restart_interval: 1 2024-09-10T18:57:53.280 INFO:journalctl@ceph.mon.b.smithi194.stdout: metadata_block_size: 4096 2024-09-10T18:57:53.281 INFO:journalctl@ceph.mon.b.smithi194.stdout: partition_filters: 0 2024-09-10T18:57:53.281 INFO:journalctl@ceph.mon.b.smithi194.stdout: use_delta_encoding: 1 2024-09-10T18:57:53.281 INFO:journalctl@ceph.mon.b.smithi194.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-10T18:57:53.281 INFO:journalctl@ceph.mon.b.smithi194.stdout: whole_key_filtering: 1 2024-09-10T18:57:53.281 INFO:journalctl@ceph.mon.b.smithi194.stdout: verify_compression: 0 2024-09-10T18:57:53.281 INFO:journalctl@ceph.mon.b.smithi194.stdout: read_amp_bytes_per_bit: 0 2024-09-10T18:57:53.281 INFO:journalctl@ceph.mon.b.smithi194.stdout: format_version: 4 2024-09-10T18:57:53.281 INFO:journalctl@ceph.mon.b.smithi194.stdout: enable_index_compression: 1 2024-09-10T18:57:53.281 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_align: 0 2024-09-10T18:57:53.281 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.write_buffer_size: 33554432 2024-09-10T18:57:53.281 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_write_buffer_number: 2 2024-09-10T18:57:53.281 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compression: NoCompression 2024-09-10T18:57:53.281 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.bottommost_compression: Disabled 2024-09-10T18:57:53.281 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.prefix_extractor: nullptr 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.num_levels: 7 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compression_opts.level: 32767 2024-09-10T18:57:53.282 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compression_opts.strategy: 0 2024-09-10T18:57:53.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-10T18:57:53.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:57:53.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-10T18:57:53.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compression_opts.enabled: false 2024-09-10T18:57:53.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-10T18:57:53.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-10T18:57:53.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-10T18:57:53.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.target_file_size_base: 67108864 2024-09-10T18:57:53.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-10T18:57:53.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-10T18:57:53.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-10T18:57:53.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-10T18:57:53.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-10T18:57:53.283 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-10T18:57:53.284 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-10T18:57:53.284 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-10T18:57:53.284 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-10T18:57:53.284 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-10T18:57:53.284 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-10T18:57:53.284 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-10T18:57:53.284 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-10T18:57:53.284 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.arena_block_size: 4194304 2024-09-10T18:57:53.284 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-10T18:57:53.284 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-10T18:57:53.284 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-10T18:57:53.284 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.disable_auto_compactions: 0 2024-09-10T18:57:53.284 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-10T18:57:53.285 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-10T18:57:53.285 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-10T18:57:53.285 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-10T18:57:53.285 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-10T18:57:53.285 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-10T18:57:53.285 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-10T18:57:53.285 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-10T18:57:53.285 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-10T18:57:53.285 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-10T18:57:53.285 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-10T18:57:53.285 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.inplace_update_support: 0 2024-09-10T18:57:53.285 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-10T18:57:53.286 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-10T18:57:53.286 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-10T18:57:53.286 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-10T18:57:53.286 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.bloom_locality: 0 2024-09-10T18:57:53.286 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.max_successive_merges: 0 2024-09-10T18:57:53.286 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-10T18:57:53.286 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.paranoid_file_checks: 0 2024-09-10T18:57:53.286 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.force_consistency_checks: 1 2024-09-10T18:57:53.286 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.report_bg_io_stats: 0 2024-09-10T18:57:53.286 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.ttl: 2592000 2024-09-10T18:57:53.286 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-10T18:57:53.286 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.enable_blob_files: false 2024-09-10T18:57:53.286 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.min_blob_size: 0 2024-09-10T18:57:53.287 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.blob_file_size: 268435456 2024-09-10T18:57:53.287 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-10T18:57:53.287 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-10T18:57:53.287 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-10T18:57:53.287 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-10T18:57:53.287 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-10T18:57:53.287 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-10T18:57:53.287 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725994672813177, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-10T18:57:53.287 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-10T18:57:53.287 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-10T18:57:53.287 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725994672813478, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1571, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1725994672, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "7928c95a-6e87-4b0f-bc6c-fb818096c813", "db_session_id": "8YRQFNDTVF0GQ03BZMF1"}} 2024-09-10T18:57:53.287 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-10T18:57:53.287 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725994672813798, "job": 1, "event": "recovery_finished"} 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55cc43490700 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: DB pointer 0x55cc43460000 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** DB Stats ** 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-09-10T18:57:53.288 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** Compaction Stats [default] ** 2024-09-10T18:57:53.289 INFO:journalctl@ceph.mon.b.smithi194.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:57:53.289 INFO:journalctl@ceph.mon.b.smithi194.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:57:53.289 INFO:journalctl@ceph.mon.b.smithi194.stdout: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-10T18:57:53.289 INFO:journalctl@ceph.mon.b.smithi194.stdout: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-10T18:57:53.289 INFO:journalctl@ceph.mon.b.smithi194.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-10T18:57:53.289 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-09-10T18:57:53.289 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** Compaction Stats [default] ** 2024-09-10T18:57:53.289 INFO:journalctl@ceph.mon.b.smithi194.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:57:53.289 INFO:journalctl@ceph.mon.b.smithi194.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:57:53.289 INFO:journalctl@ceph.mon.b.smithi194.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-10T18:57:53.289 INFO:journalctl@ceph.mon.b.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:57:53.289 INFO:journalctl@ceph.mon.b.smithi194.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:57:53.289 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** Compaction Stats [default] ** 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: L0 1/0 1.53 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: Sum 1/0 1.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-10T18:57:53.290 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** Compaction Stats [default] ** 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.2 0.00 0.00 1 0.000 0 0 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-09-10T18:57:53.291 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: using public_addr v2:172.21.15.194:0/0 -> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: starting mon.b rank -1 at public addrs [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] at bind addrs [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b@-1(???) e0 preinit fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b@-1(synchronizing).mds e1 new map 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b@-1(synchronizing).mds e1 print_map 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout: e1 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout: legacy client fscid: -1 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout: No filesystems configured 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-10T18:57:53.292 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-10T18:57:53.293 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-10T18:57:53.293 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-10T18:57:53.293 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-10T18:57:53.293 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-10T18:57:53.293 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:57:53.293 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:57:53.293 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:57:53.293 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-10T18:57:53.293 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:57:53.293 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2446877516' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2446877516' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:57:53.294 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:57:53.295 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi194:172.21.15.194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:57:53.295 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: Saving service mon spec with placement smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi194:172.21.15.194=b;count:3 2024-09-10T18:57:53.295 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: Deploying daemon mon.b on smithi194 2024-09-10T18:57:53.295 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:53.295 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:53.295 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/1010097481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:57:53.295 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 ceph-mon[32464]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-10T18:57:53.295 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 bash[32425]: 8762862dea5c36023485aa02040f92a7c6ce35c0d06943310900f8dd840dbbf7 2024-09-10T18:57:53.295 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:52 smithi194 systemd[1]: Started Ceph mon.b for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T18:57:53.405 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-10T18:57:53.405 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T18:57:53.661 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:57:58.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.178+0000 7fc28263f640 1 -- 172.21.15.194:0/733783267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc27c102310 msgr2=0x7fc264005540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:58.181 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.178+0000 7fc28263f640 1 --2- 172.21.15.194:0/733783267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc27c102310 0x7fc264005540 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc26c007d80 tx=0x7fc26c00b040 comp rx=0 tx=0).stop 2024-09-10T18:57:58.181 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.178+0000 7fc28263f640 1 -- 172.21.15.194:0/733783267 shutdown_connections 2024-09-10T18:57:58.181 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.178+0000 7fc28263f640 1 --2- 172.21.15.194:0/733783267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc27c102310 0x7fc264005540 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:58.181 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.178+0000 7fc28263f640 1 -- 172.21.15.194:0/733783267 >> 172.21.15.194:0/733783267 conn(0x7fc27c0f7ec0 msgr2=0x7fc27c0fa2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:58.181 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.178+0000 7fc28263f640 1 -- 172.21.15.194:0/733783267 shutdown_connections 2024-09-10T18:57:58.181 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.178+0000 7fc28263f640 1 -- 172.21.15.194:0/733783267 wait complete. 2024-09-10T18:57:58.181 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:57:58.181 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:57:58.181 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: mon.a calling monitor election 2024-09-10T18:57:58.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:57:58.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:58.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:57:58.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: mon.b calling monitor election 2024-09-10T18:57:58.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:57:58.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:58.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:57:58.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:57:58.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-10T18:57:58.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: monmap e2: 2 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:57:58.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: fsmap 2024-09-10T18:57:58.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:57:58.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: mgrmap e13: y(active, since 39s) 2024-09-10T18:57:58.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: overall HEALTH_OK 2024-09-10T18:57:58.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:58.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:58.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:58.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:57:58.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:57 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:57:58.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.179+0000 7fc28263f640 1 Processor -- start 2024-09-10T18:57:58.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.179+0000 7fc28263f640 1 -- start start 2024-09-10T18:57:58.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc28263f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc27c102310 0x7fc27c114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:58.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc28263f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc27c1145a0 0x7fc27c10e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:58.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc28263f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27c114b20 con 0x7fc27c102310 2024-09-10T18:57:58.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc28263f640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27c114c60 con 0x7fc27c1145a0 2024-09-10T18:57:58.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc27b7fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc27c1145a0 0x7fc27c10e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:58.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc27bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc27c102310 0x7fc27c114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:58.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc27bfff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc27c102310 0x7fc27c114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.194:44256/0 (socket says 172.21.15.194:44256) 2024-09-10T18:57:58.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc27b7fe640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc27c1145a0 0x7fc27c10e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:42378/0 (socket says 172.21.15.194:42378) 2024-09-10T18:57:58.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc27bfff640 1 -- 172.21.15.194:0/2420759040 learned_addr learned my addr 172.21.15.194:0/2420759040 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T18:57:58.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc27b7fe640 1 -- 172.21.15.194:0/2420759040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc27c1145a0 msgr2=0x7fc27c10e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T18:57:58.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc27b7fe640 1 -- 172.21.15.194:0/2420759040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc27c1145a0 msgr2=0x7fc27c10e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T18:57:58.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc27b7fe640 1 --2- 172.21.15.194:0/2420759040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc27c1145a0 0x7fc27c10e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-10T18:57:58.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc27b7fe640 1 --2- 172.21.15.194:0/2420759040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc27c1145a0 0x7fc27c10e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T18:57:58.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc27bfff640 1 -- 172.21.15.194:0/2420759040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc27c1145a0 msgr2=0x7fc27c10e030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T18:57:58.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc27bfff640 1 --2- 172.21.15.194:0/2420759040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc27c1145a0 0x7fc27c10e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:58.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc27bfff640 1 -- 172.21.15.194:0/2420759040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c005040 con 0x7fc27c102310 2024-09-10T18:57:58.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.180+0000 7fc27bfff640 1 --2- 172.21.15.194:0/2420759040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc27c102310 0x7fc27c114060 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc26c031940 tx=0x7fc26c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:58.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.181+0000 7fc278ff9640 1 -- 172.21.15.194:0/2420759040 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fc26c03a420 con 0x7fc27c102310 2024-09-10T18:57:58.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.181+0000 7fc278ff9640 1 -- 172.21.15.194:0/2420759040 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fc26c03e070 con 0x7fc27c102310 2024-09-10T18:57:58.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.181+0000 7fc28263f640 1 -- 172.21.15.194:0/2420759040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc27c10e5d0 con 0x7fc27c102310 2024-09-10T18:57:58.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.181+0000 7fc278ff9640 1 -- 172.21.15.194:0/2420759040 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fc26c044620 con 0x7fc27c102310 2024-09-10T18:57:58.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.181+0000 7fc28263f640 1 -- 172.21.15.194:0/2420759040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc27c10eb70 con 0x7fc27c102310 2024-09-10T18:57:58.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.182+0000 7fc278ff9640 1 -- 172.21.15.194:0/2420759040 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 45514+0+0 (secure 0 0 0) 0x7fc26c0448e0 con 0x7fc27c102310 2024-09-10T18:57:58.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.182+0000 7fc28263f640 1 -- 172.21.15.194:0/2420759040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc27c10ee40 con 0x7fc27c102310 2024-09-10T18:57:58.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.182+0000 7fc278ff9640 1 --2- 172.21.15.194:0/2420759040 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc268039640 0x7fc26803baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:57:58.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.182+0000 7fc278ff9640 1 -- 172.21.15.194:0/2420759040 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc26c06dc50 con 0x7fc27c102310 2024-09-10T18:57:58.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.182+0000 7fc27b7fe640 1 --2- 172.21.15.194:0/2420759040 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc268039640 0x7fc26803baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:57:58.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.182+0000 7fc27b7fe640 1 --2- 172.21.15.194:0/2420759040 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc268039640 0x7fc26803baf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc270006fd0 tx=0x7fc270008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:57:58.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.187+0000 7fc278ff9640 1 -- 172.21.15.194:0/2420759040 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fc26c030e20 con 0x7fc27c102310 2024-09-10T18:57:58.288 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:57:58.288 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:57:58.288 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: mon.a calling monitor election 2024-09-10T18:57:58.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:57:58.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:58.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:57:58.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: mon.b calling monitor election 2024-09-10T18:57:58.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:57:58.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:58.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:57:58.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:57:58.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-10T18:57:58.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: monmap e2: 2 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:57:58.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: fsmap 2024-09-10T18:57:58.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:57:58.289 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: mgrmap e13: y(active, since 39s) 2024-09-10T18:57:58.290 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: overall HEALTH_OK 2024-09-10T18:57:58.290 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:58.290 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:58.290 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:57:58.290 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:57:58.290 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:57 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:57:58.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.412+0000 7fc28263f640 1 -- 172.21.15.194:0/2420759040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc27c0fda80 con 0x7fc27c102310 2024-09-10T18:57:58.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.413+0000 7fc278ff9640 1 -- 172.21.15.194:0/2420759040 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1005 (secure 0 0 0) 0x7fc26c040090 con 0x7fc27c102310 2024-09-10T18:57:58.415 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:57:58.415 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":2,"fsid":"3afe9804-6fa6-11ef-bcea-c7b262605968","modified":"2024-09-10T18:57:52.840459Z","created":"2024-09-10T18:56:23.443244Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-10T18:57:58.415 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 2 2024-09-10T18:57:58.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.415+0000 7fc28263f640 1 -- 172.21.15.194:0/2420759040 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc268039640 msgr2=0x7fc26803baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:58.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.415+0000 7fc28263f640 1 --2- 172.21.15.194:0/2420759040 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc268039640 0x7fc26803baf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc270006fd0 tx=0x7fc270008040 comp rx=0 tx=0).stop 2024-09-10T18:57:58.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.415+0000 7fc28263f640 1 -- 172.21.15.194:0/2420759040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc27c102310 msgr2=0x7fc27c114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:57:58.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.415+0000 7fc28263f640 1 --2- 172.21.15.194:0/2420759040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc27c102310 0x7fc27c114060 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc26c031940 tx=0x7fc26c00b040 comp rx=0 tx=0).stop 2024-09-10T18:57:58.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.415+0000 7fc28263f640 1 -- 172.21.15.194:0/2420759040 shutdown_connections 2024-09-10T18:57:58.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.415+0000 7fc28263f640 1 --2- 172.21.15.194:0/2420759040 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc268039640 0x7fc26803baf0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:58.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.415+0000 7fc28263f640 1 --2- 172.21.15.194:0/2420759040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc27c1145a0 0x7fc27c10e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:58.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.415+0000 7fc28263f640 1 --2- 172.21.15.194:0/2420759040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc27c102310 0x7fc27c114060 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:57:58.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.415+0000 7fc28263f640 1 -- 172.21.15.194:0/2420759040 >> 172.21.15.194:0/2420759040 conn(0x7fc27c0f7ec0 msgr2=0x7fc27c0ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:57:58.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.415+0000 7fc28263f640 1 -- 172.21.15.194:0/2420759040 shutdown_connections 2024-09-10T18:57:58.417 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:57:58.415+0000 7fc28263f640 1 -- 172.21.15.194:0/2420759040 wait complete. 2024-09-10T18:57:59.207 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:58 smithi194 ceph-mon[32464]: Deploying daemon mon.c on smithi119 2024-09-10T18:57:59.207 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:58 smithi194 ceph-mon[32464]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:59.207 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/2420759040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:57:59.207 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:57:58 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:57:59.225 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:58 smithi119 ceph-mon[29577]: Deploying daemon mon.c on smithi119 2024-09-10T18:57:59.225 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:58 smithi119 ceph-mon[29577]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:57:59.226 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/2420759040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:57:59.226 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:57:58 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:57:59.970 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-10T18:57:59.971 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T18:58:00.210 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:58:01.059 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 podman[40636]: 2024-09-10 18:58:01.057961368 +0000 UTC m=+0.169116337 container create d7b0fe23492836672bccf895fcda7ee533c3817bfc0d209658d81cb4d519ce4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-10T18:58:01.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:01 smithi194 ceph-mon[32464]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:01.309 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[29577]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:01.309 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 podman[40636]: 2024-09-10 18:58:01.285442742 +0000 UTC m=+0.396597668 container init d7b0fe23492836672bccf895fcda7ee533c3817bfc0d209658d81cb4d519ce4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-c, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-10T18:58:01.309 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 podman[40636]: 2024-09-10 18:58:01.289925661 +0000 UTC m=+0.401080592 container start d7b0fe23492836672bccf895fcda7ee533c3817bfc0d209658d81cb4d519ce4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-c, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:58:01.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: set uid:gid to 167:167 (ceph:ceph) 2024-09-10T18:58:01.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable), process ceph-mon, pid 7 2024-09-10T18:58:01.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: pidfile_write: ignore empty --pid-file 2024-09-10T18:58:01.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: load: jerasure load: lrc 2024-09-10T18:58:01.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: RocksDB version: 6.15.5 2024-09-10T18:58:01.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-10T18:58:01.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Compile date Sep 9 2024 2024-09-10T18:58:01.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: DB SUMMARY 2024-09-10T18:58:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: DB Session ID: YL6SKDSU7KSV5VU0JV7A 2024-09-10T18:58:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: CURRENT file: CURRENT 2024-09-10T18:58:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: IDENTITY file: IDENTITY 2024-09-10T18:58:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-10T18:58:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-10T18:58:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-10T18:58:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.error_if_exists: 0 2024-09-10T18:58:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.create_if_missing: 0 2024-09-10T18:58:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.paranoid_checks: 1 2024-09-10T18:58:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-10T18:58:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.env: 0x562d4849f440 2024-09-10T18:58:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.fs: Posix File System 2024-09-10T18:58:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.info_log: 0x562d49592ec0 2024-09-10T18:58:01.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_file_opening_threads: 16 2024-09-10T18:58:01.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.statistics: (nil) 2024-09-10T18:58:01.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.use_fsync: 0 2024-09-10T18:58:01.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_log_file_size: 0 2024-09-10T18:58:01.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-10T18:58:01.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-10T18:58:01.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.keep_log_file_num: 1000 2024-09-10T18:58:01.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.recycle_log_file_num: 0 2024-09-10T18:58:01.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.allow_fallocate: 1 2024-09-10T18:58:01.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.allow_mmap_reads: 0 2024-09-10T18:58:01.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.allow_mmap_writes: 0 2024-09-10T18:58:01.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.use_direct_reads: 0 2024-09-10T18:58:01.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-10T18:58:01.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.create_missing_column_families: 0 2024-09-10T18:58:01.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.db_log_dir: 2024-09-10T18:58:01.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-09-10T18:58:01.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-10T18:58:01.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-10T18:58:01.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-10T18:58:01.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-10T18:58:01.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-10T18:58:01.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-10T18:58:01.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.advise_random_on_open: 1 2024-09-10T18:58:01.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.db_write_buffer_size: 0 2024-09-10T18:58:01.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.write_buffer_manager: 0x562d496507b0 2024-09-10T18:58:01.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-10T18:58:01.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-10T18:58:01.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-10T18:58:01.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-10T18:58:01.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.rate_limiter: (nil) 2024-09-10T18:58:01.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-10T18:58:01.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.wal_recovery_mode: 2 2024-09-10T18:58:01.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.enable_thread_tracking: 0 2024-09-10T18:58:01.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.enable_pipelined_write: 0 2024-09-10T18:58:01.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.unordered_write: 0 2024-09-10T18:58:01.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-10T18:58:01.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-10T18:58:01.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-10T18:58:01.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-10T18:58:01.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.row_cache: None 2024-09-10T18:58:01.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.wal_filter: None 2024-09-10T18:58:01.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-10T18:58:01.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.allow_ingest_behind: 0 2024-09-10T18:58:01.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.preserve_deletes: 0 2024-09-10T18:58:01.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.two_write_queues: 0 2024-09-10T18:58:01.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.manual_wal_flush: 0 2024-09-10T18:58:01.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.atomic_flush: 0 2024-09-10T18:58:01.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-10T18:58:01.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-10T18:58:01.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-10T18:58:01.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.log_readahead_size: 0 2024-09-10T18:58:01.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-10T18:58:01.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.best_efforts_recovery: 0 2024-09-10T18:58:01.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-10T18:58:01.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-10T18:58:01.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.allow_data_in_errors: 0 2024-09-10T18:58:01.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.db_host_id: __hostname__ 2024-09-10T18:58:01.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_background_jobs: 2 2024-09-10T18:58:01.574 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_background_compactions: -1 2024-09-10T18:58:01.574 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_subcompactions: 1 2024-09-10T18:58:01.574 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-10T18:58:01.575 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-10T18:58:01.575 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-10T18:58:01.575 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_total_wal_size: 0 2024-09-10T18:58:01.575 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-10T18:58:01.575 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-10T18:58:01.575 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-10T18:58:01.575 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-10T18:58:01.575 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_open_files: -1 2024-09-10T18:58:01.575 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.bytes_per_sync: 0 2024-09-10T18:58:01.575 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-10T18:58:01.575 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-10T18:58:01.575 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compaction_readahead_size: 0 2024-09-10T18:58:01.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_background_flushes: -1 2024-09-10T18:58:01.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Compression algorithms supported: 2024-09-10T18:58:01.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: kZSTD supported: 0 2024-09-10T18:58:01.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: kXpressCompression supported: 0 2024-09-10T18:58:01.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: kBZip2Compression supported: 0 2024-09-10T18:58:01.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-10T18:58:01.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: kLZ4Compression supported: 1 2024-09-10T18:58:01.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: kZlibCompression supported: 1 2024-09-10T18:58:01.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: kLZ4HCCompression supported: 1 2024-09-10T18:58:01.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: kSnappyCompression supported: 1 2024-09-10T18:58:01.576 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-10T18:58:01.577 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-09-10T18:58:01.577 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-10T18:58:01.577 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-10T18:58:01.577 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.merge_operator: 2024-09-10T18:58:01.577 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compaction_filter: None 2024-09-10T18:58:01.577 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compaction_filter_factory: None 2024-09-10T18:58:01.577 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.sst_partitioner_factory: None 2024-09-10T18:58:01.577 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-10T18:58:01.577 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-10T18:58:01.577 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562d49539a38) 2024-09-10T18:58:01.577 INFO:journalctl@ceph.mon.c.smithi119.stdout: cache_index_and_filter_blocks: 1 2024-09-10T18:58:01.577 INFO:journalctl@ceph.mon.c.smithi119.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-10T18:58:01.577 INFO:journalctl@ceph.mon.c.smithi119.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-10T18:58:01.577 INFO:journalctl@ceph.mon.c.smithi119.stdout: pin_top_level_index_and_filter: 1 2024-09-10T18:58:01.578 INFO:journalctl@ceph.mon.c.smithi119.stdout: index_type: 0 2024-09-10T18:58:01.578 INFO:journalctl@ceph.mon.c.smithi119.stdout: data_block_index_type: 0 2024-09-10T18:58:01.578 INFO:journalctl@ceph.mon.c.smithi119.stdout: index_shortening: 1 2024-09-10T18:58:01.578 INFO:journalctl@ceph.mon.c.smithi119.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-10T18:58:01.578 INFO:journalctl@ceph.mon.c.smithi119.stdout: hash_index_allow_collision: 1 2024-09-10T18:58:01.578 INFO:journalctl@ceph.mon.c.smithi119.stdout: checksum: 1 2024-09-10T18:58:01.578 INFO:journalctl@ceph.mon.c.smithi119.stdout: no_block_cache: 0 2024-09-10T18:58:01.579 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_cache: 0x562d4956f1f0 2024-09-10T18:58:01.579 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_cache_name: BinnedLRUCache 2024-09-10T18:58:01.579 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_cache_options: 2024-09-10T18:58:01.579 INFO:journalctl@ceph.mon.c.smithi119.stdout: capacity : 536870912 2024-09-10T18:58:01.579 INFO:journalctl@ceph.mon.c.smithi119.stdout: num_shard_bits : 4 2024-09-10T18:58:01.579 INFO:journalctl@ceph.mon.c.smithi119.stdout: strict_capacity_limit : 0 2024-09-10T18:58:01.579 INFO:journalctl@ceph.mon.c.smithi119.stdout: high_pri_pool_ratio: 0.000 2024-09-10T18:58:01.579 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_cache_compressed: (nil) 2024-09-10T18:58:01.579 INFO:journalctl@ceph.mon.c.smithi119.stdout: persistent_cache: (nil) 2024-09-10T18:58:01.579 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_size: 4096 2024-09-10T18:58:01.579 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_size_deviation: 10 2024-09-10T18:58:01.579 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_restart_interval: 16 2024-09-10T18:58:01.579 INFO:journalctl@ceph.mon.c.smithi119.stdout: index_block_restart_interval: 1 2024-09-10T18:58:01.579 INFO:journalctl@ceph.mon.c.smithi119.stdout: metadata_block_size: 4096 2024-09-10T18:58:01.580 INFO:journalctl@ceph.mon.c.smithi119.stdout: partition_filters: 0 2024-09-10T18:58:01.580 INFO:journalctl@ceph.mon.c.smithi119.stdout: use_delta_encoding: 1 2024-09-10T18:58:01.580 INFO:journalctl@ceph.mon.c.smithi119.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-10T18:58:01.580 INFO:journalctl@ceph.mon.c.smithi119.stdout: whole_key_filtering: 1 2024-09-10T18:58:01.580 INFO:journalctl@ceph.mon.c.smithi119.stdout: verify_compression: 0 2024-09-10T18:58:01.580 INFO:journalctl@ceph.mon.c.smithi119.stdout: read_amp_bytes_per_bit: 0 2024-09-10T18:58:01.580 INFO:journalctl@ceph.mon.c.smithi119.stdout: format_version: 4 2024-09-10T18:58:01.580 INFO:journalctl@ceph.mon.c.smithi119.stdout: enable_index_compression: 1 2024-09-10T18:58:01.580 INFO:journalctl@ceph.mon.c.smithi119.stdout: block_align: 0 2024-09-10T18:58:01.580 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.write_buffer_size: 33554432 2024-09-10T18:58:01.580 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_write_buffer_number: 2 2024-09-10T18:58:01.580 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compression: NoCompression 2024-09-10T18:58:01.580 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.bottommost_compression: Disabled 2024-09-10T18:58:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.prefix_extractor: nullptr 2024-09-10T18:58:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-10T18:58:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.num_levels: 7 2024-09-10T18:58:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-10T18:58:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-10T18:58:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-10T18:58:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-10T18:58:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-10T18:58:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-10T18:58:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-10T18:58:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:58:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-10T18:58:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-10T18:58:01.581 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-10T18:58:01.582 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compression_opts.level: 32767 2024-09-10T18:58:01.582 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compression_opts.strategy: 0 2024-09-10T18:58:01.582 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-10T18:58:01.582 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-10T18:58:01.582 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-10T18:58:01.582 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compression_opts.enabled: false 2024-09-10T18:58:01.582 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-10T18:58:01.582 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-10T18:58:01.582 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-10T18:58:01.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.target_file_size_base: 67108864 2024-09-10T18:58:01.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-10T18:58:01.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-10T18:58:01.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-10T18:58:01.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-10T18:58:01.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-10T18:58:01.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-10T18:58:01.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-10T18:58:01.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-10T18:58:01.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-10T18:58:01.583 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-10T18:58:01.584 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-10T18:58:01.584 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-10T18:58:01.584 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-10T18:58:01.584 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.arena_block_size: 4194304 2024-09-10T18:58:01.584 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-10T18:58:01.584 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-10T18:58:01.584 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-10T18:58:01.585 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.disable_auto_compactions: 0 2024-09-10T18:58:01.585 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-10T18:58:01.585 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-10T18:58:01.585 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-10T18:58:01.585 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-10T18:58:01.585 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-10T18:58:01.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-10T18:58:01.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-10T18:58:01.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-10T18:58:01.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-10T18:58:01.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-10T18:58:01.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-10T18:58:01.586 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.inplace_update_support: 0 2024-09-10T18:58:01.587 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-10T18:58:01.587 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-10T18:58:01.587 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-10T18:58:01.587 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-10T18:58:01.587 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.bloom_locality: 0 2024-09-10T18:58:01.587 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.max_successive_merges: 0 2024-09-10T18:58:01.587 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-10T18:58:01.588 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.paranoid_file_checks: 0 2024-09-10T18:58:01.588 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.force_consistency_checks: 1 2024-09-10T18:58:01.588 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.report_bg_io_stats: 0 2024-09-10T18:58:01.588 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.ttl: 2592000 2024-09-10T18:58:01.588 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-10T18:58:01.588 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.enable_blob_files: false 2024-09-10T18:58:01.588 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.min_blob_size: 0 2024-09-10T18:58:01.588 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.blob_file_size: 268435456 2024-09-10T18:58:01.589 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-10T18:58:01.589 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-10T18:58:01.589 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-10T18:58:01.589 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-10T18:58:01.589 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-10T18:58:01.589 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-10T18:58:01.589 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725994681313705, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-10T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-10T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-10T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725994681314007, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1696, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1725994681, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "f111a727-4ca7-4626-a64c-e914555fee65", "db_session_id": "YL6SKDSU7KSV5VU0JV7A"}} 2024-09-10T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-10T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725994681314336, "job": 1, "event": "recovery_finished"} 2024-09-10T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-10T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x562d495a2700 2024-09-10T18:58:01.590 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: DB pointer 0x562d49572000 2024-09-10T18:58:01.591 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-10T18:58:01.591 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-10T18:58:01.591 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** DB Stats ** 2024-09-10T18:58:01.591 INFO:journalctl@ceph.mon.c.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:58:01.591 INFO:journalctl@ceph.mon.c.smithi119.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-10T18:58:01.591 INFO:journalctl@ceph.mon.c.smithi119.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-10T18:58:01.591 INFO:journalctl@ceph.mon.c.smithi119.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:58:01.591 INFO:journalctl@ceph.mon.c.smithi119.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-10T18:58:01.591 INFO:journalctl@ceph.mon.c.smithi119.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-10T18:58:01.591 INFO:journalctl@ceph.mon.c.smithi119.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T18:58:01.591 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-10T18:58:01.591 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-10T18:58:01.592 INFO:journalctl@ceph.mon.c.smithi119.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:58:01.592 INFO:journalctl@ceph.mon.c.smithi119.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:58:01.592 INFO:journalctl@ceph.mon.c.smithi119.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-09-10T18:58:01.592 INFO:journalctl@ceph.mon.c.smithi119.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-09-10T18:58:01.592 INFO:journalctl@ceph.mon.c.smithi119.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-09-10T18:58:01.592 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-10T18:58:01.592 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-10T18:58:01.592 INFO:journalctl@ceph.mon.c.smithi119.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:58:01.592 INFO:journalctl@ceph.mon.c.smithi119.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:58:01.592 INFO:journalctl@ceph.mon.c.smithi119.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-09-10T18:58:01.593 INFO:journalctl@ceph.mon.c.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:58:01.593 INFO:journalctl@ceph.mon.c.smithi119.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:58:01.593 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:58:01.593 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:58:01.593 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:58:01.593 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:58:01.593 INFO:journalctl@ceph.mon.c.smithi119.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:58:01.593 INFO:journalctl@ceph.mon.c.smithi119.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:58:01.593 INFO:journalctl@ceph.mon.c.smithi119.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:58:01.593 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-10T18:58:01.593 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:58:01.594 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-10T18:58:01.594 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-10T18:58:01.594 INFO:journalctl@ceph.mon.c.smithi119.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:58:01.594 INFO:journalctl@ceph.mon.c.smithi119.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:58:01.594 INFO:journalctl@ceph.mon.c.smithi119.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-09-10T18:58:01.594 INFO:journalctl@ceph.mon.c.smithi119.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-09-10T18:58:01.594 INFO:journalctl@ceph.mon.c.smithi119.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-10T18:58:01.594 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-10T18:58:01.594 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** Compaction Stats [default] ** 2024-09-10T18:58:01.595 INFO:journalctl@ceph.mon.c.smithi119.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T18:58:01.595 INFO:journalctl@ceph.mon.c.smithi119.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T18:58:01.595 INFO:journalctl@ceph.mon.c.smithi119.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.4 0.00 0.00 1 0.000 0 0 2024-09-10T18:58:01.595 INFO:journalctl@ceph.mon.c.smithi119.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T18:58:01.595 INFO:journalctl@ceph.mon.c.smithi119.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T18:58:01.595 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T18:58:01.595 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T18:58:01.595 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T18:58:01.595 INFO:journalctl@ceph.mon.c.smithi119.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T18:58:01.596 INFO:journalctl@ceph.mon.c.smithi119.stdout: Cumulative compaction: 0.00 GB write, 0.61 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:58:01.596 INFO:journalctl@ceph.mon.c.smithi119.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T18:58:01.596 INFO:journalctl@ceph.mon.c.smithi119.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T18:58:01.596 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-10T18:58:01.596 INFO:journalctl@ceph.mon.c.smithi119.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T18:58:01.596 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-10T18:58:01.596 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: using public_addrv [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] 2024-09-10T18:58:01.596 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: starting mon.c rank -1 at public addrs [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] at bind addrs [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:58:01.596 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c@-1(???) e0 preinit fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:58:01.596 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c@-1(synchronizing).mds e1 new map 2024-09-10T18:58:01.597 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-10T18:58:01.597 INFO:journalctl@ceph.mon.c.smithi119.stdout: e1 2024-09-10T18:58:01.597 INFO:journalctl@ceph.mon.c.smithi119.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-10T18:58:01.597 INFO:journalctl@ceph.mon.c.smithi119.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-10T18:58:01.597 INFO:journalctl@ceph.mon.c.smithi119.stdout: legacy client fscid: -1 2024-09-10T18:58:01.597 INFO:journalctl@ceph.mon.c.smithi119.stdout: 2024-09-10T18:58:01.597 INFO:journalctl@ceph.mon.c.smithi119.stdout: No filesystems configured 2024-09-10T18:58:01.597 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-10T18:58:01.598 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-10T18:58:01.598 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-10T18:58:01.598 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-10T18:58:01.598 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-10T18:58:01.598 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-10T18:58:01.598 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-10T18:58:01.598 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:58:01.599 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:58:01.599 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T18:58:01.599 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-10T18:58:01.599 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:58:01.599 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:58:01.599 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T18:58:01.599 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:01.600 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:01.600 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:01.600 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2446877516' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-10T18:58:01.600 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:01.600 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2446877516' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-10T18:58:01.600 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:58:01.600 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:01.600 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:01.600 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:01.600 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:01.601 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:01.601 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:01.601 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:58:01.601 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:01.601 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi194:172.21.15.194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:58:01.601 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: Saving service mon spec with placement smithi119:172.21.15.119=a;smithi119:[v2:172.21.15.119:3301,v1:172.21.15.119:6790]=c;smithi194:172.21.15.194=b;count:3 2024-09-10T18:58:01.601 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: Deploying daemon mon.b on smithi194 2024-09-10T18:58:01.601 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:01.601 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:01.602 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/1010097481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:58:01.602 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:58:01.602 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:58:01.602 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.a calling monitor election 2024-09-10T18:58:01.602 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:58:01.602 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:01.603 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:58:01.603 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.b calling monitor election 2024-09-10T18:58:01.603 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:58:01.603 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:01.603 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:58:01.603 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:58:01.604 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-10T18:58:01.604 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: monmap e2: 2 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:58:01.604 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: fsmap 2024-09-10T18:58:01.604 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:58:01.604 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mgrmap e13: y(active, since 39s) 2024-09-10T18:58:01.605 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: overall HEALTH_OK 2024-09-10T18:58:01.605 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:01.605 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:01.605 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:01.605 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:58:01.605 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:01.605 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: Deploying daemon mon.c on smithi119 2024-09-10T18:58:01.605 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:01.605 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/2420759040' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:58:01.606 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:58:01.606 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:01.606 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 ceph-mon[40674]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-10T18:58:01.606 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 bash[40636]: d7b0fe23492836672bccf895fcda7ee533c3817bfc0d209658d81cb4d519ce4f 2024-09-10T18:58:01.606 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:01 smithi119 systemd[1]: Started Ceph mon.c for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T18:58:06.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:58:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: mon.a calling monitor election 2024-09-10T18:58:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: mon.b calling monitor election 2024-09-10T18:58:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:58:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: mon.c calling monitor election 2024-09-10T18:58:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-10T18:58:06.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0],c=[v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:58:06.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: fsmap 2024-09-10T18:58:06.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:58:06.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: mgrmap e13: y(active, since 48s) 2024-09-10T18:58:06.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: overall HEALTH_OK 2024-09-10T18:58:06.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:06.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:06.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:06.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:06 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:58:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: mon.a calling monitor election 2024-09-10T18:58:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: mon.b calling monitor election 2024-09-10T18:58:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:58:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: mon.c calling monitor election 2024-09-10T18:58:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-10T18:58:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0],c=[v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:58:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: fsmap 2024-09-10T18:58:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:58:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: mgrmap e13: y(active, since 48s) 2024-09-10T18:58:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: overall HEALTH_OK 2024-09-10T18:58:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:58:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: mon.a calling monitor election 2024-09-10T18:58:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: mon.b calling monitor election 2024-09-10T18:58:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:58:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: mon.c calling monitor election 2024-09-10T18:58:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-10T18:58:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: monmap e3: 3 mons at {a=[v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0],b=[v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0],c=[v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T18:58:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: fsmap 2024-09-10T18:58:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T18:58:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: mgrmap e13: y(active, since 48s) 2024-09-10T18:58:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: overall HEALTH_OK 2024-09-10T18:58:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:06 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:07.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.654+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/2450385509 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e10006160 msgr2=0x7f0e100038e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:07.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.654+0000 7f0e2c2d8640 1 --2- 172.21.15.194:0/2450385509 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e10006160 0x7f0e100038e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0e240fc2e0 tx=0x7f0e18002040 comp rx=0 tx=0).stop 2024-09-10T18:58:07.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.654+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/2450385509 shutdown_connections 2024-09-10T18:58:07.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.654+0000 7f0e2c2d8640 1 --2- 172.21.15.194:0/2450385509 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e10006160 0x7f0e100038e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:07.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.654+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/2450385509 >> 172.21.15.194:0/2450385509 conn(0x7f0e240f6e00 msgr2=0x7f0e240f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:07.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.654+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/2450385509 shutdown_connections 2024-09-10T18:58:07.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.654+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/2450385509 wait complete. 2024-09-10T18:58:07.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2c2d8640 1 Processor -- start 2024-09-10T18:58:07.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2c2d8640 1 -- start start 2024-09-10T18:58:07.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2c2d8640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0e10006160 0x7f0e24114170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:07.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2c2d8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e241146b0 0x7f0e2410e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:07.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2c2d8640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e2410e630 0x7f0e2410eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:07.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2c2d8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e24114d10 con 0x7f0e241146b0 2024-09-10T18:58:07.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2c2d8640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2411afc0 con 0x7f0e2410e630 2024-09-10T18:58:07.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2c2d8640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2411b100 con 0x7f0e10006160 2024-09-10T18:58:07.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e29873640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e241146b0 0x7f0e2410e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:07.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2a074640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0e10006160 0x7f0e24114170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:07.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e29873640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e241146b0 0x7f0e2410e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.194:40944/0 (socket says 172.21.15.194:40944) 2024-09-10T18:58:07.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e29873640 1 -- 172.21.15.194:0/3738228378 learned_addr learned my addr 172.21.15.194:0/3738228378 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T18:58:07.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2a875640 1 --2- 172.21.15.194:0/3738228378 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e2410e630 0x7f0e2410eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:07.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2a074640 1 -- 172.21.15.194:0/3738228378 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0e10006160 msgr2=0x7f0e24114170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-09-10T18:58:07.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2a074640 1 -- 172.21.15.194:0/3738228378 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0e10006160 msgr2=0x7f0e24114170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T18:58:07.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2a074640 1 --2- 172.21.15.194:0/3738228378 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0e10006160 0x7f0e24114170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-10T18:58:07.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2a875640 1 -- 172.21.15.194:0/3738228378 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e2410e630 msgr2=0x7f0e2410eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-10T18:58:07.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2a875640 1 -- 172.21.15.194:0/3738228378 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e2410e630 msgr2=0x7f0e2410eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T18:58:07.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2a074640 1 --2- 172.21.15.194:0/3738228378 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0e10006160 0x7f0e24114170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T18:58:07.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2a875640 1 --2- 172.21.15.194:0/3738228378 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e2410e630 0x7f0e2410eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-09-10T18:58:07.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e2a875640 1 --2- 172.21.15.194:0/3738228378 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e2410e630 0x7f0e2410eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T18:58:07.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e29873640 1 -- 172.21.15.194:0/3738228378 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0e10006160 msgr2=0x7f0e24114170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T18:58:07.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e29873640 1 --2- 172.21.15.194:0/3738228378 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0e10006160 0x7f0e24114170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:07.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e29873640 1 -- 172.21.15.194:0/3738228378 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e2410e630 msgr2=0x7f0e2410eaa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T18:58:07.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e29873640 1 --2- 172.21.15.194:0/3738228378 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e2410e630 0x7f0e2410eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:07.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.655+0000 7f0e29873640 1 -- 172.21.15.194:0/3738228378 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e18005040 con 0x7f0e241146b0 2024-09-10T18:58:07.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.656+0000 7f0e29873640 1 --2- 172.21.15.194:0/3738228378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e241146b0 0x7f0e2410e0f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000b810 tx=0x7f0e200091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:58:07.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.656+0000 7f0e16ffd640 1 -- 172.21.15.194:0/3738228378 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2000c730 con 0x7f0e241146b0 2024-09-10T18:58:07.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.656+0000 7f0e16ffd640 1 -- 172.21.15.194:0/3738228378 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f0e20009240 con 0x7f0e241146b0 2024-09-10T18:58:07.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.656+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/3738228378 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2411b390 con 0x7f0e241146b0 2024-09-10T18:58:07.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.656+0000 7f0e16ffd640 1 -- 172.21.15.194:0/3738228378 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e20014b90 con 0x7f0e241146b0 2024-09-10T18:58:07.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.656+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/3738228378 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2411b930 con 0x7f0e241146b0 2024-09-10T18:58:07.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.657+0000 7f0e16ffd640 1 -- 172.21.15.194:0/3738228378 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 13) v1 ==== 45514+0+0 (secure 0 0 0) 0x7f0e200133e0 con 0x7f0e241146b0 2024-09-10T18:58:07.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.657+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/3738228378 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e240fc960 con 0x7f0e241146b0 2024-09-10T18:58:07.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.657+0000 7f0e16ffd640 1 --2- 172.21.15.194:0/3738228378 >> v2:172.21.15.119:6800/1094129222 conn(0x7f0e08039990 0x7f0e0803be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:07.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.657+0000 7f0e16ffd640 1 -- 172.21.15.194:0/3738228378 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0e2004b5f0 con 0x7f0e241146b0 2024-09-10T18:58:07.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.657+0000 7f0e2a074640 1 --2- 172.21.15.194:0/3738228378 >> v2:172.21.15.119:6800/1094129222 conn(0x7f0e08039990 0x7f0e0803be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:07.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.658+0000 7f0e2a074640 1 --2- 172.21.15.194:0/3738228378 >> v2:172.21.15.119:6800/1094129222 conn(0x7f0e08039990 0x7f0e0803be40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009e10 tx=0x7f0e1802d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:58:07.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.662+0000 7f0e16ffd640 1 -- 172.21.15.194:0/3738228378 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f0e2001a830 con 0x7f0e241146b0 2024-09-10T18:58:07.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:07 smithi194 ceph-mon[32464]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:07.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:07 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:07.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:07 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:07.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:07 smithi119 ceph-mon[29577]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:07.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:07 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:07.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:07 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:07 smithi119 ceph-mon[40674]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:07 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:07 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:58:07.891 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.890+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/3738228378 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e2405ed70 con 0x7f0e241146b0 2024-09-10T18:58:07.892 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.890+0000 7f0e16ffd640 1 -- 172.21.15.194:0/3738228378 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1275 (secure 0 0 0) 0x7f0e2001a830 con 0x7f0e241146b0 2024-09-10T18:58:07.892 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-09-10T18:58:07.892 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":3,"fsid":"3afe9804-6fa6-11ef-bcea-c7b262605968","modified":"2024-09-10T18:58:01.335638Z","created":"2024-09-10T18:56:23.443244Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3300","nonce":0},{"type":"v1","addr":"172.21.15.119:6789","nonce":0}]},"addr":"172.21.15.119:6789/0","public_addr":"172.21.15.119:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:3301","nonce":0},{"type":"v1","addr":"172.21.15.119:6790","nonce":0}]},"addr":"172.21.15.119:6790/0","public_addr":"172.21.15.119:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-10T18:58:07.892 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 3 2024-09-10T18:58:07.894 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.892+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/3738228378 >> v2:172.21.15.119:6800/1094129222 conn(0x7f0e08039990 msgr2=0x7f0e0803be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:07.894 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.892+0000 7f0e2c2d8640 1 --2- 172.21.15.194:0/3738228378 >> v2:172.21.15.119:6800/1094129222 conn(0x7f0e08039990 0x7f0e0803be40 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009e10 tx=0x7f0e1802d040 comp rx=0 tx=0).stop 2024-09-10T18:58:07.894 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.892+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/3738228378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e241146b0 msgr2=0x7f0e2410e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:07.894 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.892+0000 7f0e2c2d8640 1 --2- 172.21.15.194:0/3738228378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e241146b0 0x7f0e2410e0f0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000b810 tx=0x7f0e200091d0 comp rx=0 tx=0).stop 2024-09-10T18:58:07.894 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.892+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/3738228378 shutdown_connections 2024-09-10T18:58:07.894 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.892+0000 7f0e2c2d8640 1 --2- 172.21.15.194:0/3738228378 >> v2:172.21.15.119:6800/1094129222 conn(0x7f0e08039990 0x7f0e0803be40 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:07.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.893+0000 7f0e2c2d8640 1 --2- 172.21.15.194:0/3738228378 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e2410e630 0x7f0e2410eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:07.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.893+0000 7f0e2c2d8640 1 --2- 172.21.15.194:0/3738228378 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e241146b0 0x7f0e2410e0f0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:07.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.893+0000 7f0e2c2d8640 1 --2- 172.21.15.194:0/3738228378 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0e10006160 0x7f0e24114170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:07.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.893+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/3738228378 >> 172.21.15.194:0/3738228378 conn(0x7f0e240f6e00 msgr2=0x7f0e24065770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:07.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.893+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/3738228378 shutdown_connections 2024-09-10T18:58:07.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:07.893+0000 7f0e2c2d8640 1 -- 172.21.15.194:0/3738228378 wait complete. 2024-09-10T18:58:08.535 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-10T18:58:08.535 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph config generate-minimal-conf 2024-09-10T18:58:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:08.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:08.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-10T18:58:08.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-10T18:58:08.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/3738228378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:58:08.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.771 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.771 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.771 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.771 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.771 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:58:08.771 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:58:08.772 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:08.772 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:08.794 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:08.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-10T18:58:08.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-10T18:58:08.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/3738228378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:58:08.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:58:08.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:58:08.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:08.795 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:08.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:08.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-10T18:58:08.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-10T18:58:08.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/3738228378' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T18:58:08.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.796 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.797 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.797 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:08.797 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:58:08.797 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:58:08.797 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:08.797 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:09.440 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[29577]: Updating smithi119:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:58:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[29577]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:58:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[29577]: Reconfiguring mon.a (unknown last config time)... 2024-09-10T18:58:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[29577]: Reconfiguring daemon mon.a on smithi119 2024-09-10T18:58:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[29577]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:58:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:58:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[40674]: Updating smithi119:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:58:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[40674]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:58:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[40674]: Reconfiguring mon.a (unknown last config time)... 2024-09-10T18:58:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[40674]: Reconfiguring daemon mon.a on smithi119 2024-09-10T18:58:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[40674]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:58:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:58:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:09 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:10.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:09 smithi194 ceph-mon[32464]: Updating smithi119:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:58:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:09 smithi194 ceph-mon[32464]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T18:58:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:09 smithi194 ceph-mon[32464]: Reconfiguring mon.a (unknown last config time)... 2024-09-10T18:58:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:09 smithi194 ceph-mon[32464]: Reconfiguring daemon mon.a on smithi119 2024-09-10T18:58:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:09 smithi194 ceph-mon[32464]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:09 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:09 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:09 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:58:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:09 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:58:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:09 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:10 smithi119 ceph-mon[29577]: Reconfiguring mon.c (monmap changed)... 2024-09-10T18:58:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:10 smithi119 ceph-mon[29577]: Reconfiguring daemon mon.c on smithi119 2024-09-10T18:58:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:10 smithi119 ceph-mon[40674]: Reconfiguring mon.c (monmap changed)... 2024-09-10T18:58:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:10 smithi119 ceph-mon[40674]: Reconfiguring daemon mon.c on smithi119 2024-09-10T18:58:11.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:10 smithi194 ceph-mon[32464]: Reconfiguring mon.c (monmap changed)... 2024-09-10T18:58:11.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:10 smithi194 ceph-mon[32464]: Reconfiguring daemon mon.c on smithi119 2024-09-10T18:58:11.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.503+0000 7f0418ac9640 1 -- 172.21.15.119:0/2481927145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0414066910 msgr2=0x7f0414109680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:11.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.503+0000 7f0418ac9640 1 --2- 172.21.15.119:0/2481927145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0414066910 0x7f0414109680 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f040c009cd0 tx=0x7f040c004d10 comp rx=0 tx=0).stop 2024-09-10T18:58:11.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.503+0000 7f0418ac9640 1 -- 172.21.15.119:0/2481927145 shutdown_connections 2024-09-10T18:58:11.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.503+0000 7f0418ac9640 1 --2- 172.21.15.119:0/2481927145 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0414066910 0x7f0414109680 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:11.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.503+0000 7f0418ac9640 1 --2- 172.21.15.119:0/2481927145 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0414065f80 0x7f04140663d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:11.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.503+0000 7f0418ac9640 1 --2- 172.21.15.119:0/2481927145 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0414102980 0x7f0414102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:11.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.503+0000 7f0418ac9640 1 -- 172.21.15.119:0/2481927145 >> 172.21.15.119:0/2481927145 conn(0x7f04140724d0 msgr2=0x7f04140728d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:11.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.503+0000 7f0418ac9640 1 -- 172.21.15.119:0/2481927145 shutdown_connections 2024-09-10T18:58:11.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.503+0000 7f0418ac9640 1 -- 172.21.15.119:0/2481927145 wait complete. 2024-09-10T18:58:11.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.503+0000 7f0418ac9640 1 Processor -- start 2024-09-10T18:58:11.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.503+0000 7f0418ac9640 1 -- start start 2024-09-10T18:58:11.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.503+0000 7f0418ac9640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0414065f80 0x7f0414112c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:11.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f0418ac9640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0414066910 0x7f0414113150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:11.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f0418ac9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0414102980 0x7f041410cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:11.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f0418ac9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041410d3f0 con 0x7f0414102980 2024-09-10T18:58:11.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f0418ac9640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f041410d560 con 0x7f0414066910 2024-09-10T18:58:11.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f0418ac9640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f041410d720 con 0x7f0414065f80 2024-09-10T18:58:11.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f041259c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0414065f80 0x7f0414112c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:11.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f041259c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0414065f80 0x7f0414112c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:49298/0 (socket says 172.21.15.119:49298) 2024-09-10T18:58:11.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f041259c640 1 -- 172.21.15.119:0/1039058769 learned_addr learned my addr 172.21.15.119:0/1039058769 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:58:11.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f0412d9d640 1 --2- 172.21.15.119:0/1039058769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0414102980 0x7f041410cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:11.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f0411d9b640 1 --2- 172.21.15.119:0/1039058769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0414066910 0x7f0414113150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:11.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f041259c640 1 -- 172.21.15.119:0/1039058769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0414066910 msgr2=0x7f0414113150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:11.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f041259c640 1 --2- 172.21.15.119:0/1039058769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0414066910 0x7f0414113150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:11.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f041259c640 1 -- 172.21.15.119:0/1039058769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0414102980 msgr2=0x7f041410cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:11.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f041259c640 1 --2- 172.21.15.119:0/1039058769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0414102980 0x7f041410cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:11.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f041259c640 1 -- 172.21.15.119:0/1039058769 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0404005040 con 0x7f0414065f80 2024-09-10T18:58:11.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f0412d9d640 1 --2- 172.21.15.119:0/1039058769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0414102980 0x7f041410cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T18:58:11.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f041259c640 1 --2- 172.21.15.119:0/1039058769 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0414065f80 0x7f0414112c10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0404009780 tx=0x7f0404009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:58:11.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f0402ffd640 1 -- 172.21.15.119:0/1039058769 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0404007c30 con 0x7f0414065f80 2024-09-10T18:58:11.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f0402ffd640 1 -- 172.21.15.119:0/1039058769 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f040400f040 con 0x7f0414065f80 2024-09-10T18:58:11.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f0418ac9640 1 -- 172.21.15.119:0/1039058769 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040c0098b0 con 0x7f0414065f80 2024-09-10T18:58:11.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f0402ffd640 1 -- 172.21.15.119:0/1039058769 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0404013b60 con 0x7f0414065f80 2024-09-10T18:58:11.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.504+0000 7f0418ac9640 1 -- 172.21.15.119:0/1039058769 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0414119a60 con 0x7f0414065f80 2024-09-10T18:58:11.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.505+0000 7f0402ffd640 1 -- 172.21.15.119:0/1039058769 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 13) v1 ==== 45514+0+0 (secure 0 0 0) 0x7f040401c410 con 0x7f0414065f80 2024-09-10T18:58:11.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.505+0000 7f0402ffd640 1 --2- 172.21.15.119:0/1039058769 >> v2:172.21.15.119:6800/1094129222 conn(0x7f03fc039750 0x7f03fc03bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:11.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.505+0000 7f0402ffd640 1 -- 172.21.15.119:0/1039058769 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f040404bc60 con 0x7f0414065f80 2024-09-10T18:58:11.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.505+0000 7f0411d9b640 1 --2- 172.21.15.119:0/1039058769 >> v2:172.21.15.119:6800/1094129222 conn(0x7f03fc039750 0x7f03fc03bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:11.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.505+0000 7f0418ac9640 1 -- 172.21.15.119:0/1039058769 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04140fe060 con 0x7f0414065f80 2024-09-10T18:58:11.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.506+0000 7f0411d9b640 1 --2- 172.21.15.119:0/1039058769 >> v2:172.21.15.119:6800/1094129222 conn(0x7f03fc039750 0x7f03fc03bc00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f041410e4d0 tx=0x7f0408004330 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:58:11.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.510+0000 7f0402ffd640 1 -- 172.21.15.119:0/1039058769 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f0404022860 con 0x7f0414065f80 2024-09-10T18:58:11.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.671+0000 7f0418ac9640 1 -- 172.21.15.119:0/1039058769 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f041410be70 con 0x7f0414065f80 2024-09-10T18:58:11.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.671+0000 7f0402ffd640 1 -- 172.21.15.119:0/1039058769 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f0404022290 con 0x7f0414065f80 2024-09-10T18:58:11.673 INFO:teuthology.orchestra.run.smithi119.stdout:# minimal ceph.conf for 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:58:11.673 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2024-09-10T18:58:11.673 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T18:58:11.673 INFO:teuthology.orchestra.run.smithi119.stdout: mon_host = [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] 2024-09-10T18:58:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.673+0000 7f0418ac9640 1 -- 172.21.15.119:0/1039058769 >> v2:172.21.15.119:6800/1094129222 conn(0x7f03fc039750 msgr2=0x7f03fc03bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.673+0000 7f0418ac9640 1 --2- 172.21.15.119:0/1039058769 >> v2:172.21.15.119:6800/1094129222 conn(0x7f03fc039750 0x7f03fc03bc00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f041410e4d0 tx=0x7f0408004330 comp rx=0 tx=0).stop 2024-09-10T18:58:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.673+0000 7f0418ac9640 1 -- 172.21.15.119:0/1039058769 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0414065f80 msgr2=0x7f0414112c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.673+0000 7f0418ac9640 1 --2- 172.21.15.119:0/1039058769 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0414065f80 0x7f0414112c10 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f0404009780 tx=0x7f0404009da0 comp rx=0 tx=0).stop 2024-09-10T18:58:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.673+0000 7f0418ac9640 1 -- 172.21.15.119:0/1039058769 shutdown_connections 2024-09-10T18:58:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.673+0000 7f0418ac9640 1 --2- 172.21.15.119:0/1039058769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0414102980 0x7f041410cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.673+0000 7f0418ac9640 1 --2- 172.21.15.119:0/1039058769 >> v2:172.21.15.119:6800/1094129222 conn(0x7f03fc039750 0x7f03fc03bc00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.673+0000 7f0418ac9640 1 --2- 172.21.15.119:0/1039058769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0414066910 0x7f0414113150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:11.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.673+0000 7f0418ac9640 1 --2- 172.21.15.119:0/1039058769 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0414065f80 0x7f0414112c10 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:11.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.673+0000 7f0418ac9640 1 -- 172.21.15.119:0/1039058769 >> 172.21.15.119:0/1039058769 conn(0x7f04140724d0 msgr2=0x7f04140fa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:11.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.673+0000 7f0418ac9640 1 -- 172.21.15.119:0/1039058769 shutdown_connections 2024-09-10T18:58:11.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:11.674+0000 7f0418ac9640 1 -- 172.21.15.119:0/1039058769 wait complete. 2024-09-10T18:58:11.683 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:11 smithi119 ceph-mon[29577]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:11.684 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:11 smithi119 ceph-mon[40674]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:12.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:11 smithi194 ceph-mon[32464]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:12.600 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1039058769' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:12.600 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:12 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:12.600 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:12 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:12.600 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:12 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:58:12.600 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:12 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:58:12.600 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:12 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:12.600 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1039058769' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:12.600 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:12 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:12.601 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:12 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:12.601 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:12 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:58:12.601 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:12 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:58:12.601 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:12 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:12.624 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-10T18:58:12.624 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T18:58:12.624 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.conf 2024-09-10T18:58:12.651 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T18:58:12.651 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:58:12.716 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-10T18:58:12.717 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.conf 2024-09-10T18:58:12.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1039058769' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:12.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:12 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:12.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:12 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:12.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:12 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T18:58:12.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:12 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T18:58:12.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:12 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:12.744 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-10T18:58:12.745 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T18:58:12.810 INFO:tasks.cephadm:Adding mgr.y on smithi119 2024-09-10T18:58:12.810 INFO:tasks.cephadm:Adding mgr.x on smithi194 2024-09-10T18:58:12.810 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch apply mgr '2;smithi119=y;smithi194=x' 2024-09-10T18:58:13.420 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:58:13.643 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:13 smithi194 ceph-mon[32464]: Reconfiguring mon.b (monmap changed)... 2024-09-10T18:58:13.644 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:13 smithi194 ceph-mon[32464]: Reconfiguring daemon mon.b on smithi194 2024-09-10T18:58:13.644 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:13 smithi194 ceph-mon[32464]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:13.644 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:13 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:13.644 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:13 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:13.644 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:13 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:13.644 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:13 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:13.644 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:13 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:13.644 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:13 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[29577]: Reconfiguring mon.b (monmap changed)... 2024-09-10T18:58:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[29577]: Reconfiguring daemon mon.b on smithi194 2024-09-10T18:58:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[29577]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:13.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:13.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[40674]: Reconfiguring mon.b (monmap changed)... 2024-09-10T18:58:13.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[40674]: Reconfiguring daemon mon.b on smithi194 2024-09-10T18:58:13.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[40674]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:13.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:13.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:13.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:13.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:13.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:13.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:13 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:14.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.928+0000 7f372724a640 1 -- 172.21.15.194:0/2593189625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37200ffa50 msgr2=0x7f3720101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:14.933 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.928+0000 7f372724a640 1 --2- 172.21.15.194:0/2593189625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37200ffa50 0x7f3720101e30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3714004950 tx=0x7f3714009da0 comp rx=0 tx=0).stop 2024-09-10T18:58:14.933 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.928+0000 7f372724a640 1 -- 172.21.15.194:0/2593189625 shutdown_connections 2024-09-10T18:58:14.933 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.928+0000 7f372724a640 1 --2- 172.21.15.194:0/2593189625 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f37201024a0 0x7f3720104950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:14.933 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.928+0000 7f372724a640 1 --2- 172.21.15.194:0/2593189625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37200ffa50 0x7f3720101e30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:14.933 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.928+0000 7f372724a640 1 --2- 172.21.15.194:0/2593189625 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3720066b40 0x7f37200ff510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:14.933 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.928+0000 7f372724a640 1 -- 172.21.15.194:0/2593189625 >> 172.21.15.194:0/2593189625 conn(0x7f37200f6ee0 msgr2=0x7f37200f9330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:14.933 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.928+0000 7f372724a640 1 -- 172.21.15.194:0/2593189625 shutdown_connections 2024-09-10T18:58:14.933 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.928+0000 7f372724a640 1 -- 172.21.15.194:0/2593189625 wait complete. 2024-09-10T18:58:14.933 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.928+0000 7f372724a640 1 Processor -- start 2024-09-10T18:58:14.933 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.928+0000 7f372724a640 1 -- start start 2024-09-10T18:58:14.933 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.928+0000 7f372724a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3720066b40 0x7f37201132f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:14.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.928+0000 7f372724a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f37200ffa50 0x7f3720113830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:14.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.928+0000 7f372724a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37201024a0 0x7f372011a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:14.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f372724a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372011ae60 con 0x7f3720066b40 2024-09-10T18:58:14.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f372724a640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372011afd0 con 0x7f37201024a0 2024-09-10T18:58:14.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f372724a640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372011b190 con 0x7f37200ffa50 2024-09-10T18:58:14.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f37257e7640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37201024a0 0x7f372011a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:14.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f37257e7640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37201024a0 0x7f372011a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:55596/0 (socket says 172.21.15.194:55596) 2024-09-10T18:58:14.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f37257e7640 1 -- 172.21.15.194:0/3339099067 learned_addr learned my addr 172.21.15.194:0/3339099067 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T18:58:14.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f3724fe6640 1 --2- 172.21.15.194:0/3339099067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3720066b40 0x7f37201132f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:14.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f371ffff640 1 --2- 172.21.15.194:0/3339099067 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f37200ffa50 0x7f3720113830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:14.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f37257e7640 1 -- 172.21.15.194:0/3339099067 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f37200ffa50 msgr2=0x7f3720113830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:14.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f37257e7640 1 --2- 172.21.15.194:0/3339099067 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f37200ffa50 0x7f3720113830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:14.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f37257e7640 1 -- 172.21.15.194:0/3339099067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3720066b40 msgr2=0x7f37201132f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:14.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f37257e7640 1 --2- 172.21.15.194:0/3339099067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3720066b40 0x7f37201132f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:14.935 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f37257e7640 1 -- 172.21.15.194:0/3339099067 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3714005040 con 0x7f37201024a0 2024-09-10T18:58:14.935 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f37257e7640 1 --2- 172.21.15.194:0/3339099067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37201024a0 0x7f372011a890 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f371800c960 tx=0x7f3718007ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:58:14.935 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f371d7fa640 1 -- 172.21.15.194:0/3339099067 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718004020 con 0x7f37201024a0 2024-09-10T18:58:14.935 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f372724a640 1 -- 172.21.15.194:0/3339099067 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372011b470 con 0x7f37201024a0 2024-09-10T18:58:14.935 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f371d7fa640 1 -- 172.21.15.194:0/3339099067 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f3718004180 con 0x7f37201024a0 2024-09-10T18:58:14.935 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f371d7fa640 1 -- 172.21.15.194:0/3339099067 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37180038e0 con 0x7f37201024a0 2024-09-10T18:58:14.935 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.929+0000 7f372724a640 1 -- 172.21.15.194:0/3339099067 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372011b9c0 con 0x7f37201024a0 2024-09-10T18:58:14.935 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.930+0000 7f371d7fa640 1 -- 172.21.15.194:0/3339099067 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) v1 ==== 45514+0+0 (secure 0 0 0) 0x7f3718003cd0 con 0x7f37201024a0 2024-09-10T18:58:14.935 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.930+0000 7f372724a640 1 -- 172.21.15.194:0/3339099067 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f372004bf60 con 0x7f37201024a0 2024-09-10T18:58:14.935 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.930+0000 7f371d7fa640 1 --2- 172.21.15.194:0/3339099067 >> v2:172.21.15.119:6800/1094129222 conn(0x7f3708039770 0x7f370803bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:14.935 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.930+0000 7f371d7fa640 1 -- 172.21.15.194:0/3339099067 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f371804c220 con 0x7f37201024a0 2024-09-10T18:58:14.935 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.931+0000 7f3724fe6640 1 --2- 172.21.15.194:0/3339099067 >> v2:172.21.15.119:6800/1094129222 conn(0x7f3708039770 0x7f370803bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:14.935 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.931+0000 7f3724fe6640 1 --2- 172.21.15.194:0/3339099067 >> v2:172.21.15.119:6800/1094129222 conn(0x7f3708039770 0x7f370803bc20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f37200668a0 tx=0x7f3710009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:58:14.936 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:14.935+0000 7f371d7fa640 1 -- 172.21.15.194:0/3339099067 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f37180228f0 con 0x7f37201024a0 2024-09-10T18:58:15.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:15.097+0000 7f372724a640 1 -- 172.21.15.194:0/3339099067 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi119=y;smithi194=x", "target": ["mon-mgr", ""]}) v1 -- 0x7f37200fd2a0 con 0x7f3708039770 2024-09-10T18:58:15.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:15.103+0000 7f371d7fa640 1 -- 172.21.15.194:0/3339099067 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f37200fd2a0 con 0x7f3708039770 2024-09-10T18:58:15.105 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mgr update... 2024-09-10T18:58:15.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:15.105+0000 7f372724a640 1 -- 172.21.15.194:0/3339099067 >> v2:172.21.15.119:6800/1094129222 conn(0x7f3708039770 msgr2=0x7f370803bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:15.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:15.105+0000 7f372724a640 1 --2- 172.21.15.194:0/3339099067 >> v2:172.21.15.119:6800/1094129222 conn(0x7f3708039770 0x7f370803bc20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f37200668a0 tx=0x7f3710009040 comp rx=0 tx=0).stop 2024-09-10T18:58:15.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:15.105+0000 7f372724a640 1 -- 172.21.15.194:0/3339099067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37201024a0 msgr2=0x7f372011a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:15.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:15.105+0000 7f372724a640 1 --2- 172.21.15.194:0/3339099067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37201024a0 0x7f372011a890 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f371800c960 tx=0x7f3718007ec0 comp rx=0 tx=0).stop 2024-09-10T18:58:15.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:15.105+0000 7f372724a640 1 -- 172.21.15.194:0/3339099067 shutdown_connections 2024-09-10T18:58:15.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:15.105+0000 7f372724a640 1 --2- 172.21.15.194:0/3339099067 >> v2:172.21.15.119:6800/1094129222 conn(0x7f3708039770 0x7f370803bc20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:15.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:15.105+0000 7f372724a640 1 --2- 172.21.15.194:0/3339099067 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f37201024a0 0x7f372011a890 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:15.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:15.105+0000 7f372724a640 1 --2- 172.21.15.194:0/3339099067 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f37200ffa50 0x7f3720113830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:15.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:15.105+0000 7f372724a640 1 --2- 172.21.15.194:0/3339099067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3720066b40 0x7f37201132f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:15.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:15.105+0000 7f372724a640 1 -- 172.21.15.194:0/3339099067 >> 172.21.15.194:0/3339099067 conn(0x7f37200f6ee0 msgr2=0x7f37200fdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:15.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:15.105+0000 7f372724a640 1 -- 172.21.15.194:0/3339099067 shutdown_connections 2024-09-10T18:58:15.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:58:15.105+0000 7f372724a640 1 -- 172.21.15.194:0/3339099067 wait complete. 2024-09-10T18:58:15.652 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:15 smithi194 ceph-mon[32464]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:15.653 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:15 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:15.653 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:15 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:15.653 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:15 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:15.653 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:15 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:15.653 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:15 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:15.653 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:15 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:58:15.653 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:15 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-10T18:58:15.653 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:15 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:58:15.653 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:15 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:15.676 DEBUG:teuthology.orchestra.run.smithi194:mgr.x> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mgr.x.service 2024-09-10T18:58:15.678 INFO:tasks.cephadm:Deploying OSDs... 2024-09-10T18:58:15.679 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T18:58:15.679 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T18:58:15.699 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T18:58:15.699 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2024-09-10T18:58:15.757 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T18:58:15.757 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:58:15.758 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 967 Links: 1 2024-09-10T18:58:15.758 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:58:15.758 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:58:15.758 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-10 18:58:00.737436811 +0000 2024-09-10T18:58:15.758 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-10 18:53:06.299793458 +0000 2024-09-10T18:58:15.758 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-10 18:53:06.299793458 +0000 2024-09-10T18:58:15.758 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-10 18:53:06.299793458 +0000 2024-09-10T18:58:15.758 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T18:58:15.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[29577]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:58:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-10T18:58:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:58:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:15.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[40674]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:15.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:15.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:15.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:15.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:15.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:15.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:58:15.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-10T18:58:15.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:58:15.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:15 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:15.825 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-10T18:58:15.825 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-10T18:58:15.825 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000295086 s, 1.7 MB/s 2024-09-10T18:58:15.827 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T18:58:15.886 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2024-09-10T18:58:15.944 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T18:58:15.944 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:58:15.944 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 971 Links: 1 2024-09-10T18:58:15.944 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:58:15.944 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:58:15.944 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-10 18:58:00.738436786 +0000 2024-09-10T18:58:15.944 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-10 18:53:06.299793458 +0000 2024-09-10T18:58:15.944 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-10 18:53:06.299793458 +0000 2024-09-10T18:58:15.945 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-10 18:53:06.299793458 +0000 2024-09-10T18:58:15.945 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T18:58:16.011 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-10T18:58:16.011 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-10T18:58:16.011 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000211209 s, 2.4 MB/s 2024-09-10T18:58:16.012 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T18:58:16.070 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2024-09-10T18:58:16.127 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T18:58:16.127 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:58:16.127 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 982 Links: 1 2024-09-10T18:58:16.127 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:58:16.127 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:58:16.127 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-10 18:58:00.738436786 +0000 2024-09-10T18:58:16.127 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-10 18:53:06.300793433 +0000 2024-09-10T18:58:16.127 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-10 18:53:06.300793433 +0000 2024-09-10T18:58:16.127 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-10 18:53:06.300793433 +0000 2024-09-10T18:58:16.128 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T18:58:16.192 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-10T18:58:16.192 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-10T18:58:16.192 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000190811 s, 2.7 MB/s 2024-09-10T18:58:16.194 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T18:58:16.251 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2024-09-10T18:58:16.307 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T18:58:16.307 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:58:16.307 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-10T18:58:16.307 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:58:16.307 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:58:16.307 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-10 18:58:00.738436786 +0000 2024-09-10T18:58:16.307 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-10 18:53:06.299793458 +0000 2024-09-10T18:58:16.308 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-10 18:53:06.299793458 +0000 2024-09-10T18:58:16.308 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: 2024-09-10 18:53:06.299793458 +0000 2024-09-10T18:58:16.308 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T18:58:16.371 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-10T18:58:16.372 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-10T18:58:16.372 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000215947 s, 2.4 MB/s 2024-09-10T18:58:16.373 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T18:58:16.429 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-10T18:58:16.430 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T18:58:16.447 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T18:58:16.447 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-09-10T18:58:16.503 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T18:58:16.503 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:58:16.503 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 975 Links: 1 2024-09-10T18:58:16.503 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:58:16.503 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:58:16.503 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-10 18:57:52.099824783 +0000 2024-09-10T18:58:16.504 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-10 18:53:06.329460423 +0000 2024-09-10T18:58:16.504 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-10 18:53:06.329460423 +0000 2024-09-10T18:58:16.504 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-09-10 18:53:06.329460423 +0000 2024-09-10T18:58:16.504 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T18:58:16.567 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-10T18:58:16.567 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-10T18:58:16.567 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.00018852 s, 2.7 MB/s 2024-09-10T18:58:16.569 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T18:58:16.625 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-09-10T18:58:16.680 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T18:58:16.680 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:58:16.681 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 984 Links: 1 2024-09-10T18:58:16.681 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:58:16.681 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:58:16.681 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-10 18:57:52.100824763 +0000 2024-09-10T18:58:16.681 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-10 18:53:06.330460404 +0000 2024-09-10T18:58:16.681 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-10 18:53:06.330460404 +0000 2024-09-10T18:58:16.681 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-09-10 18:53:06.330460404 +0000 2024-09-10T18:58:16.681 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T18:58:16.747 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-10T18:58:16.747 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-10T18:58:16.747 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000234307 s, 2.2 MB/s 2024-09-10T18:58:16.748 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T18:58:16.804 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-09-10T18:58:16.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:16 smithi119 ceph-mon[29577]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi119=y;smithi194=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:58:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:16 smithi119 ceph-mon[29577]: Saving service mgr spec with placement smithi119=y;smithi194=x;count:2 2024-09-10T18:58:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:16 smithi119 ceph-mon[29577]: Deploying daemon mgr.x on smithi194 2024-09-10T18:58:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:16 smithi119 ceph-mon[40674]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi119=y;smithi194=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:58:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:16 smithi119 ceph-mon[40674]: Saving service mgr spec with placement smithi119=y;smithi194=x;count:2 2024-09-10T18:58:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:16 smithi119 ceph-mon[40674]: Deploying daemon mgr.x on smithi194 2024-09-10T18:58:16.841 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:16 smithi194 ceph-mon[32464]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi119=y;smithi194=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:58:16.842 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:16 smithi194 ceph-mon[32464]: Saving service mgr spec with placement smithi119=y;smithi194=x;count:2 2024-09-10T18:58:16.842 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:16 smithi194 ceph-mon[32464]: Deploying daemon mgr.x on smithi194 2024-09-10T18:58:16.856 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T18:58:16.856 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:58:16.856 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 974 Links: 1 2024-09-10T18:58:16.857 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:58:16.857 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:58:16.857 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-10 18:57:52.100824763 +0000 2024-09-10T18:58:16.857 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-10 18:53:06.329460423 +0000 2024-09-10T18:58:16.857 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-10 18:53:06.329460423 +0000 2024-09-10T18:58:16.857 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-09-10 18:53:06.329460423 +0000 2024-09-10T18:58:16.857 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T18:58:17.016 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-10T18:58:17.017 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-10T18:58:17.017 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000195711 s, 2.6 MB/s 2024-09-10T18:58:17.018 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T18:58:17.034 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-09-10T18:58:17.095 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T18:58:17.095 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T18:58:17.095 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 968 Links: 1 2024-09-10T18:58:17.095 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T18:58:17.095 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T18:58:17.095 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-09-10 18:58:16.972336115 +0000 2024-09-10T18:58:17.095 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-09-10 18:53:06.329460423 +0000 2024-09-10T18:58:17.095 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-09-10 18:53:06.329460423 +0000 2024-09-10T18:58:17.095 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-09-10 18:53:06.329460423 +0000 2024-09-10T18:58:17.095 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T18:58:17.197 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-09-10T18:58:17.197 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-09-10T18:58:17.197 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000202527 s, 2.5 MB/s 2024-09-10T18:58:17.198 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T18:58:17.214 INFO:tasks.cephadm:Deploying osd.0 on smithi119 with /dev/vg_nvme/lv_4... 2024-09-10T18:58:17.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-10T18:58:17.434 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:17 smithi194 systemd[1]: Starting Ceph mgr.x for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T18:58:17.454 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:17.704 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:17 smithi194 ceph-mon[32464]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:17.705 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:17 smithi194 podman[34535]: 2024-09-10 18:58:17.335233253 +0000 UTC m=+0.015441584 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T18:58:17.705 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:17 smithi194 podman[34535]: 2024-09-10 18:58:17.438142334 +0000 UTC m=+0.118350696 container create 1f9b2bbe1ba1d2ecf98bdebaaacfb3798d667e055fc895bc436f38baa2dbcc26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T18:58:17.705 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:17 smithi194 podman[34535]: 2024-09-10 18:58:17.678269573 +0000 UTC m=+0.358477912 container init 1f9b2bbe1ba1d2ecf98bdebaaacfb3798d667e055fc895bc436f38baa2dbcc26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-10T18:58:17.705 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:17 smithi194 podman[34535]: 2024-09-10 18:58:17.682734545 +0000 UTC m=+0.362942866 container start 1f9b2bbe1ba1d2ecf98bdebaaacfb3798d667e055fc895bc436f38baa2dbcc26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-10T18:58:17.705 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:17 smithi194 ceph-mgr[34573]: -- 172.21.15.194:0/3291240778 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b208d44340 con 0x55b208d50c00 2024-09-10T18:58:17.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:17 smithi119 ceph-mon[29577]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:17.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:17 smithi119 ceph-mon[40674]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:18.018 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:17 smithi194 bash[34535]: 1f9b2bbe1ba1d2ecf98bdebaaacfb3798d667e055fc895bc436f38baa2dbcc26 2024-09-10T18:58:18.018 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:17 smithi194 systemd[1]: Started Ceph mgr.x for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T18:58:18.018 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:17 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:17.794+0000 7f29b7f0eec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T18:58:18.018 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:17 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:17.856+0000 7f29b7f0eec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T18:58:18.018 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:17 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:17.921+0000 7f29b7f0eec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T18:58:18.478 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:18.051+0000 7f29b7f0eec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:58:18.478 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:18.197+0000 7f29b7f0eec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T18:58:18.478 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:18.476+0000 7f29b7f0eec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T18:58:18.768 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:18.541+0000 7f29b7f0eec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T18:58:18.768 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:18.710+0000 7f29b7f0eec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T18:58:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:18 smithi119 ceph-mon[29577]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:18 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:18 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:18 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:18 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:18 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:19.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:18 smithi119 ceph-mon[40674]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:19.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:18 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.125 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-10T18:58:19.214 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-10T18:58:19.231 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-09-10T18:58:19.231 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-09-10T18:58:19.231 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01492 s, 703 MB/s 2024-09-10T18:58:19.231 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-09-10T18:58:19.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:18 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:18 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:18 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:18 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:18 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:18 smithi194 ceph-mon[32464]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:18 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:19.825 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi119:vg_nvme/lv_4 2024-09-10T18:58:20.018 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:19 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:19.563+0000 7f29b7f0eec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T18:58:20.018 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:19 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:19.625+0000 7f29b7f0eec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T18:58:20.416 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:20.441 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:20.328+0000 7f29b7f0eec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T18:58:20.441 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:20.439+0000 7f29b7f0eec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T18:58:20.441 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:20.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:20.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:20.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:20.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:20.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-mon[32464]: Reconfiguring mgr.y (unknown last config time)... 2024-09-10T18:58:20.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:58:20.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:58:20.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:20.442 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-mon[32464]: Reconfiguring daemon mgr.y on smithi119 2024-09-10T18:58:20.503 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:20.503 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:20.503 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:20.503 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:20.503 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:20.503 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[29577]: Reconfiguring mgr.y (unknown last config time)... 2024-09-10T18:58:20.503 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:58:20.503 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:58:20.503 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:20.503 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[29577]: Reconfiguring daemon mgr.y on smithi119 2024-09-10T18:58:20.504 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:20.504 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:20.504 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:20.504 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:20.504 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:20.504 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[40674]: Reconfiguring mgr.y (unknown last config time)... 2024-09-10T18:58:20.504 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T18:58:20.504 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T18:58:20.504 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:20.504 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:20 smithi119 ceph-mon[40674]: Reconfiguring daemon mgr.y on smithi119 2024-09-10T18:58:20.741 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:20.558+0000 7f29b7f0eec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T18:58:20.741 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:20.615+0000 7f29b7f0eec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T18:58:20.741 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:20.739+0000 7f29b7f0eec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T18:58:21.000 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:20.806+0000 7f29b7f0eec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T18:58:21.000 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:20.865+0000 7f29b7f0eec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T18:58:21.000 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:20.998+0000 7f29b7f0eec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T18:58:21.213 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:21 smithi119 ceph-mon[29577]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:21.213 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:21 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:21.213 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:21 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:21.213 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:21 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:21.213 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:21 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:21.213 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:21 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:21.214 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:21 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:21.214 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:21 smithi119 ceph-mon[40674]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:21.214 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:21 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:21.214 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:21 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:21.215 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:21 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:21.215 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:21 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:21.215 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:21 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:21.215 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:21 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:21.261 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:21 smithi194 ceph-mon[32464]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:21.261 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:21 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:21.262 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:21 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:21.262 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:21 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:21.262 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:21 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:21.262 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:21 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:21.262 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:21 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:21.262 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:21 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:21.061+0000 7f29b7f0eec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T18:58:21.262 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:21 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:21.125+0000 7f29b7f0eec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T18:58:21.262 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:21 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:21.204+0000 7f29b7f0eec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T18:58:21.262 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:21 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:21.260+0000 7f29b7f0eec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T18:58:21.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:21 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:21.316+0000 7f29b7f0eec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T18:58:22.249 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:22 smithi119 ceph-mon[29577]: Standby manager daemon x started 2024-09-10T18:58:22.249 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:22 smithi119 ceph-mon[29577]: from='mgr.? 172.21.15.194:0/1940567848' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:58:22.249 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:22 smithi119 ceph-mon[29577]: from='mgr.? 172.21.15.194:0/1940567848' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:58:22.249 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:22 smithi119 ceph-mon[29577]: from='mgr.? 172.21.15.194:0/1940567848' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:58:22.249 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:22 smithi119 ceph-mon[29577]: from='mgr.? 172.21.15.194:0/1940567848' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:58:22.249 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:22 smithi119 ceph-mon[40674]: Standby manager daemon x started 2024-09-10T18:58:22.249 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:22 smithi119 ceph-mon[40674]: from='mgr.? 172.21.15.194:0/1940567848' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:58:22.249 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:22 smithi119 ceph-mon[40674]: from='mgr.? 172.21.15.194:0/1940567848' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:58:22.249 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:22 smithi119 ceph-mon[40674]: from='mgr.? 172.21.15.194:0/1940567848' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:58:22.250 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:22 smithi119 ceph-mon[40674]: from='mgr.? 172.21.15.194:0/1940567848' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:58:22.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.247+0000 7f726375a640 1 -- 172.21.15.119:0/2254154721 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f725c0fc860 msgr2=0x7f725c0fcc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:22.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.247+0000 7f726375a640 1 --2- 172.21.15.119:0/2254154721 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f725c0fc860 0x7f725c0fcc70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7258004890 tx=0x7f7258009da0 comp rx=0 tx=0).stop 2024-09-10T18:58:22.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.247+0000 7f726375a640 1 -- 172.21.15.119:0/2254154721 shutdown_connections 2024-09-10T18:58:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.247+0000 7f726375a640 1 --2- 172.21.15.119:0/2254154721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f725c0fe4e0 0x7f725c104d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.247+0000 7f726375a640 1 --2- 172.21.15.119:0/2254154721 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f725c0fdb30 0x7f725c0fdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.247+0000 7f726375a640 1 --2- 172.21.15.119:0/2254154721 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f725c0fc860 0x7f725c0fcc70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.247+0000 7f726375a640 1 -- 172.21.15.119:0/2254154721 >> 172.21.15.119:0/2254154721 conn(0x7f725c0f8040 msgr2=0x7f725c0fa450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.247+0000 7f726375a640 1 -- 172.21.15.119:0/2254154721 shutdown_connections 2024-09-10T18:58:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.247+0000 7f726375a640 1 -- 172.21.15.119:0/2254154721 wait complete. 2024-09-10T18:58:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f726375a640 1 Processor -- start 2024-09-10T18:58:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f726375a640 1 -- start start 2024-09-10T18:58:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f726375a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f725c0fc860 0x7f725c10cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f726375a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f725c0fdb30 0x7f725c10d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f726375a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f725c0fe4e0 0x7f725c1142c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f726375a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725c114890 con 0x7f725c0fc860 2024-09-10T18:58:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f726375a640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f725c114a00 con 0x7f725c0fdb30 2024-09-10T18:58:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f726375a640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f725c114bc0 con 0x7f725c0fe4e0 2024-09-10T18:58:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f72614f6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f725c0fc860 0x7f725c10cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f7261cf7640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f725c0fe4e0 0x7f725c1142c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f72614f6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f725c0fc860 0x7f725c10cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47504/0 (socket says 172.21.15.119:47504) 2024-09-10T18:58:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f72614f6640 1 -- 172.21.15.119:0/1417738050 learned_addr learned my addr 172.21.15.119:0/1417738050 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:58:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f7260cf5640 1 --2- 172.21.15.119:0/1417738050 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f725c0fdb30 0x7f725c10d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f72614f6640 1 -- 172.21.15.119:0/1417738050 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f725c0fe4e0 msgr2=0x7f725c1142c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f72614f6640 1 --2- 172.21.15.119:0/1417738050 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f725c0fe4e0 0x7f725c1142c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f72614f6640 1 -- 172.21.15.119:0/1417738050 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f725c0fdb30 msgr2=0x7f725c10d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f72614f6640 1 --2- 172.21.15.119:0/1417738050 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f725c0fdb30 0x7f725c10d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:22.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f72614f6640 1 -- 172.21.15.119:0/1417738050 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f724c005040 con 0x7f725c0fc860 2024-09-10T18:58:22.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.248+0000 7f7261cf7640 1 --2- 172.21.15.119:0/1417738050 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f725c0fe4e0 0x7f725c1142c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T18:58:22.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.249+0000 7f72614f6640 1 --2- 172.21.15.119:0/1417738050 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f725c0fc860 0x7f725c10cd60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7258004860 tx=0x7f7258009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:58:22.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.249+0000 7f7251ffb640 1 -- 172.21.15.119:0/1417738050 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72580090d0 con 0x7f725c0fc860 2024-09-10T18:58:22.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.249+0000 7f7251ffb640 1 -- 172.21.15.119:0/1417738050 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f725803c8b0 con 0x7f725c0fc860 2024-09-10T18:58:22.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.249+0000 7f726375a640 1 -- 172.21.15.119:0/1417738050 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7258005040 con 0x7f725c0fc860 2024-09-10T18:58:22.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.249+0000 7f7251ffb640 1 -- 172.21.15.119:0/1417738050 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7258033630 con 0x7f725c0fc860 2024-09-10T18:58:22.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.249+0000 7f726375a640 1 -- 172.21.15.119:0/1417738050 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f725c115180 con 0x7f725c0fc860 2024-09-10T18:58:22.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.250+0000 7f726375a640 1 -- 172.21.15.119:0/1417738050 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f725c04bf60 con 0x7f725c0fc860 2024-09-10T18:58:22.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.250+0000 7f7251ffb640 1 -- 172.21.15.119:0/1417738050 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 14) v1 ==== 90690+0+0 (secure 0 0 0) 0x7f7258030020 con 0x7f725c0fc860 2024-09-10T18:58:22.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.251+0000 7f7251ffb640 1 --2- 172.21.15.119:0/1417738050 >> v2:172.21.15.119:6800/1094129222 conn(0x7f724806ed50 0x7f7248071200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:22.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.251+0000 7f7251ffb640 1 -- 172.21.15.119:0/1417738050 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f72580ad2a0 con 0x7f725c0fc860 2024-09-10T18:58:22.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.251+0000 7f7260cf5640 1 --2- 172.21.15.119:0/1417738050 >> v2:172.21.15.119:6800/1094129222 conn(0x7f724806ed50 0x7f7248071200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:22.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.251+0000 7f7260cf5640 1 --2- 172.21.15.119:0/1417738050 >> v2:172.21.15.119:6800/1094129222 conn(0x7f724806ed50 0x7f7248071200 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f724c007fd0 tx=0x7f724c00b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:58:22.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.255+0000 7f7251ffb640 1 -- 172.21.15.119:0/1417738050 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f725807b7e0 con 0x7f725c0fc860 2024-09-10T18:58:22.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:22.419+0000 7f726375a640 1 -- 172.21.15.119:0/1417738050 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f725c107830 con 0x7f724806ed50 2024-09-10T18:58:22.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 18:58:22 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T18:58:22.043+0000 7f29b7f0eec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T18:58:22.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:22 smithi194 ceph-mon[32464]: Standby manager daemon x started 2024-09-10T18:58:22.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:22 smithi194 ceph-mon[32464]: from='mgr.? 172.21.15.194:0/1940567848' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T18:58:22.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:22 smithi194 ceph-mon[32464]: from='mgr.? 172.21.15.194:0/1940567848' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T18:58:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:22 smithi194 ceph-mon[32464]: from='mgr.? 172.21.15.194:0/1940567848' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T18:58:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:22 smithi194 ceph-mon[32464]: from='mgr.? 172.21.15.194:0/1940567848' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T18:58:23.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:23 smithi194 ceph-mon[32464]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-10T18:58:23.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:23 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:58:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:23 smithi194 ceph-mon[32464]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:23 smithi194 ceph-mon[32464]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:58:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:23 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:58:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:23 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:58:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:23 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:23.558 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:23 smithi119 ceph-mon[29577]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-10T18:58:23.559 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:23 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:58:23.559 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:23 smithi119 ceph-mon[29577]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:23.559 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:23 smithi119 ceph-mon[29577]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:58:23.559 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:23 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:58:23.559 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:23 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:58:23.559 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:23 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:23.559 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:23 smithi119 ceph-mon[40674]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-10T18:58:23.559 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:23 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T18:58:23.559 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:23 smithi119 ceph-mon[40674]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:23.559 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:23 smithi119 ceph-mon[40674]: from='client.14211 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:58:23.559 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:23 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:58:23.559 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:23 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:58:23.560 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:23 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:25.282 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:25 smithi119 ceph-mon[29577]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:25.282 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:25 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1301686994' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c2033fa-42d6-44b1-ac93-8e4ea2aea00f"}]: dispatch 2024-09-10T18:58:25.283 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:25 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1301686994' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c2033fa-42d6-44b1-ac93-8e4ea2aea00f"}]': finished 2024-09-10T18:58:25.283 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:25 smithi119 ceph-mon[29577]: osdmap e5: 1 total, 0 up, 1 in 2024-09-10T18:58:25.283 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:25 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:25.283 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:25 smithi119 ceph-mon[40674]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:25.283 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:25 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1301686994' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c2033fa-42d6-44b1-ac93-8e4ea2aea00f"}]: dispatch 2024-09-10T18:58:25.283 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:25 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1301686994' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c2033fa-42d6-44b1-ac93-8e4ea2aea00f"}]': finished 2024-09-10T18:58:25.283 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:25 smithi119 ceph-mon[40674]: osdmap e5: 1 total, 0 up, 1 in 2024-09-10T18:58:25.283 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:25 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:25.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:25 smithi194 ceph-mon[32464]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:25.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:25 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1301686994' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7c2033fa-42d6-44b1-ac93-8e4ea2aea00f"}]: dispatch 2024-09-10T18:58:25.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:25 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1301686994' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7c2033fa-42d6-44b1-ac93-8e4ea2aea00f"}]': finished 2024-09-10T18:58:25.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:25 smithi194 ceph-mon[32464]: osdmap e5: 1 total, 0 up, 1 in 2024-09-10T18:58:25.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:25 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:26.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1104457251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:58:26.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1104457251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:58:26.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1104457251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:58:27.471 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:27 smithi119 ceph-mon[29577]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:27.472 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:27 smithi119 ceph-mon[40674]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:27.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:27 smithi194 ceph-mon[32464]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:29.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:29 smithi119 ceph-mon[29577]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:29.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:29 smithi119 ceph-mon[40674]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:29.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:29 smithi194 ceph-mon[32464]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:31.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:31 smithi194 ceph-mon[32464]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:31.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:31 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-10T18:58:31.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:31 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:31.549 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:31 smithi119 ceph-mon[29577]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:31.549 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-10T18:58:31.549 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:31.549 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:31 smithi119 ceph-mon[40674]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:31.550 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:31 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-10T18:58:31.550 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:31 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:32.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:32 smithi194 ceph-mon[32464]: Deploying daemon osd.0 on smithi119 2024-09-10T18:58:32.547 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:32 smithi119 ceph-mon[29577]: Deploying daemon osd.0 on smithi119 2024-09-10T18:58:32.547 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:32 smithi119 ceph-mon[40674]: Deploying daemon osd.0 on smithi119 2024-09-10T18:58:33.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:33 smithi119 ceph-mon[29577]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:33.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:33 smithi119 ceph-mon[40674]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:33.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:33 smithi194 ceph-mon[32464]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:35.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:35 smithi194 ceph-mon[32464]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:35.544 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:35 smithi119 ceph-mon[29577]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:35.544 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:35 smithi119 ceph-mon[40674]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:36.264 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:36 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:36.264 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:36 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:36.264 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:36 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:36.264 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:36 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:36.264 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:36 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:36.264 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:36 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:36.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:36 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:36.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:36 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:36.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:36 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:37.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:37 smithi119 ceph-mon[29577]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:37.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:37 smithi119 ceph-mon[40674]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:38.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:37 smithi194 ceph-mon[32464]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:38.921 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 0 on host 'smithi119' 2024-09-10T18:58:38.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:38.919+0000 7f7251ffb640 1 -- 172.21.15.119:0/1417738050 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f725c107830 con 0x7f724806ed50 2024-09-10T18:58:38.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:38.921+0000 7f726375a640 1 -- 172.21.15.119:0/1417738050 >> v2:172.21.15.119:6800/1094129222 conn(0x7f724806ed50 msgr2=0x7f7248071200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:38.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:38.921+0000 7f726375a640 1 --2- 172.21.15.119:0/1417738050 >> v2:172.21.15.119:6800/1094129222 conn(0x7f724806ed50 0x7f7248071200 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f724c007fd0 tx=0x7f724c00b040 comp rx=0 tx=0).stop 2024-09-10T18:58:38.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:38.921+0000 7f726375a640 1 -- 172.21.15.119:0/1417738050 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f725c0fc860 msgr2=0x7f725c10cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:38.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:38.921+0000 7f726375a640 1 --2- 172.21.15.119:0/1417738050 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f725c0fc860 0x7f725c10cd60 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7258004860 tx=0x7f7258009da0 comp rx=0 tx=0).stop 2024-09-10T18:58:38.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:38.921+0000 7f726375a640 1 -- 172.21.15.119:0/1417738050 shutdown_connections 2024-09-10T18:58:38.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:38.921+0000 7f726375a640 1 --2- 172.21.15.119:0/1417738050 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f725c0fe4e0 0x7f725c1142c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:38.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:38.921+0000 7f726375a640 1 --2- 172.21.15.119:0/1417738050 >> v2:172.21.15.119:6800/1094129222 conn(0x7f724806ed50 0x7f7248071200 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:38.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:38.921+0000 7f726375a640 1 --2- 172.21.15.119:0/1417738050 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f725c0fdb30 0x7f725c10d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:38.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:38.921+0000 7f726375a640 1 --2- 172.21.15.119:0/1417738050 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f725c0fc860 0x7f725c10cd60 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:38.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:38.921+0000 7f726375a640 1 -- 172.21.15.119:0/1417738050 >> 172.21.15.119:0/1417738050 conn(0x7f725c0f8040 msgr2=0x7f725c0f9ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:38.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:38.922+0000 7f726375a640 1 -- 172.21.15.119:0/1417738050 shutdown_connections 2024-09-10T18:58:38.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:38.922+0000 7f726375a640 1 -- 172.21.15.119:0/1417738050 wait complete. 2024-09-10T18:58:39.515 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[29577]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:39.515 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.515 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.515 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:39.515 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:39.515 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[29577]: from='osd.0 v2:172.21.15.119:6801/1598455131' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:58:39.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[29577]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:58:39.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:39.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.516 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.516 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[40674]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:39.516 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.516 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.516 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:39.516 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:39.516 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.517 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[40674]: from='osd.0 v2:172.21.15.119:6801/1598455131' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:58:39.517 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[40674]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:58:39.517 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:39.517 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.517 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:39 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:39 smithi194 ceph-mon[32464]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:39.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:39 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:39 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:39 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:39.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:39 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:39.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:39 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:39 smithi194 ceph-mon[32464]: from='osd.0 v2:172.21.15.119:6801/1598455131' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:58:39.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:39 smithi194 ceph-mon[32464]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T18:58:39.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:39 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:58:39.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:39 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:39 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:39.538 DEBUG:teuthology.orchestra.run.smithi119:osd.0> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.0.service 2024-09-10T18:58:39.540 INFO:tasks.cephadm:Deploying osd.1 on smithi119 with /dev/vg_nvme/lv_3... 2024-09-10T18:58:39.540 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-10T18:58:40.332 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:40.697 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:40 smithi119 ceph-mon[29577]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T18:58:40.697 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:40 smithi119 ceph-mon[29577]: osdmap e6: 1 total, 0 up, 1 in 2024-09-10T18:58:40.698 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:40 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:40.698 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:40 smithi119 ceph-mon[29577]: from='osd.0 v2:172.21.15.119:6801/1598455131' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:58:40.698 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:40 smithi119 ceph-mon[29577]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:58:40.698 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:40 smithi119 ceph-mon[40674]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T18:58:40.698 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:40 smithi119 ceph-mon[40674]: osdmap e6: 1 total, 0 up, 1 in 2024-09-10T18:58:40.698 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:40 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:40.698 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:40 smithi119 ceph-mon[40674]: from='osd.0 v2:172.21.15.119:6801/1598455131' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:58:40.698 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:40 smithi119 ceph-mon[40674]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:58:40.699 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 18:58:40 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0[44998]: 2024-09-10T18:58:40.610+0000 7f95586da640 -1 osd.0 0 waiting for initial osdmap 2024-09-10T18:58:40.699 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 18:58:40 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0[44998]: 2024-09-10T18:58:40.612+0000 7f9555515640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T18:58:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:40 smithi194 ceph-mon[32464]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T18:58:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:40 smithi194 ceph-mon[32464]: osdmap e6: 1 total, 0 up, 1 in 2024-09-10T18:58:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:40 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:40 smithi194 ceph-mon[32464]: from='osd.0 v2:172.21.15.119:6801/1598455131' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:58:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:40 smithi194 ceph-mon[32464]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:58:42.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:41 smithi194 ceph-mon[32464]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:41 smithi194 ceph-mon[32464]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-10T18:58:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:41 smithi194 ceph-mon[32464]: osdmap e7: 1 total, 0 up, 1 in 2024-09-10T18:58:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:41 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:41 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:42.054 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:41 smithi119 ceph-mon[29577]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:42.054 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:41 smithi119 ceph-mon[29577]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-10T18:58:42.054 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:41 smithi119 ceph-mon[29577]: osdmap e7: 1 total, 0 up, 1 in 2024-09-10T18:58:42.054 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:41 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:42.055 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:41 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:42.055 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:41 smithi119 ceph-mon[40674]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:42.055 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:41 smithi119 ceph-mon[40674]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-10T18:58:42.055 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:41 smithi119 ceph-mon[40674]: osdmap e7: 1 total, 0 up, 1 in 2024-09-10T18:58:42.055 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:41 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:42.055 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:41 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:42.892 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:42 smithi119 ceph-mon[29577]: purged_snaps scrub starts 2024-09-10T18:58:42.892 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:42 smithi119 ceph-mon[29577]: purged_snaps scrub ok 2024-09-10T18:58:42.892 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:42 smithi119 ceph-mon[29577]: osd.0 v2:172.21.15.119:6801/1598455131 boot 2024-09-10T18:58:42.893 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:42 smithi119 ceph-mon[29577]: osdmap e8: 1 total, 1 up, 1 in 2024-09-10T18:58:42.893 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:42 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:42.893 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:42 smithi119 ceph-mon[40674]: purged_snaps scrub starts 2024-09-10T18:58:42.893 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:42 smithi119 ceph-mon[40674]: purged_snaps scrub ok 2024-09-10T18:58:42.893 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:42 smithi119 ceph-mon[40674]: osd.0 v2:172.21.15.119:6801/1598455131 boot 2024-09-10T18:58:42.893 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:42 smithi119 ceph-mon[40674]: osdmap e8: 1 total, 1 up, 1 in 2024-09-10T18:58:42.893 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:42 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:43.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:42 smithi194 ceph-mon[32464]: purged_snaps scrub starts 2024-09-10T18:58:43.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:42 smithi194 ceph-mon[32464]: purged_snaps scrub ok 2024-09-10T18:58:43.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:42 smithi194 ceph-mon[32464]: osd.0 v2:172.21.15.119:6801/1598455131 boot 2024-09-10T18:58:43.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:42 smithi194 ceph-mon[32464]: osdmap e8: 1 total, 1 up, 1 in 2024-09-10T18:58:43.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:42 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T18:58:43.026 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-10T18:58:43.130 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-10T18:58:43.145 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-09-10T18:58:43.145 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-09-10T18:58:43.146 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134926 s, 777 MB/s 2024-09-10T18:58:43.146 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-09-10T18:58:43.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[40674]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:43.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[40674]: osdmap e9: 1 total, 1 up, 1 in 2024-09-10T18:58:43.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:43.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:43.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:58:43.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:43.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:43.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:43.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:43.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[29577]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:43.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[29577]: osdmap e9: 1 total, 1 up, 1 in 2024-09-10T18:58:43.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:43.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:43.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:58:43.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:43.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:43.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:43.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:44.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:43 smithi194 ceph-mon[32464]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T18:58:44.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:43 smithi194 ceph-mon[32464]: osdmap e9: 1 total, 1 up, 1 in 2024-09-10T18:58:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:43 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:43 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:43 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:58:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:43 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:43 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:43 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:58:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:43 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:58:44.209 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi119:vg_nvme/lv_3 2024-09-10T18:58:44.449 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:58:44.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:44 smithi119 ceph-mon[29577]: Detected new or changed devices on smithi119 2024-09-10T18:58:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:44 smithi119 ceph-mon[29577]: Adjusting osd_memory_target on smithi119 to 16064M 2024-09-10T18:58:44.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:44 smithi119 ceph-mon[40674]: Detected new or changed devices on smithi119 2024-09-10T18:58:44.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:44 smithi119 ceph-mon[40674]: Adjusting osd_memory_target on smithi119 to 16064M 2024-09-10T18:58:45.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:44 smithi194 ceph-mon[32464]: Detected new or changed devices on smithi119 2024-09-10T18:58:45.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:44 smithi194 ceph-mon[32464]: Adjusting osd_memory_target on smithi119 to 16064M 2024-09-10T18:58:45.858 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:45 smithi119 ceph-mon[29577]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:45.858 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:45 smithi119 ceph-mon[40674]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:46.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:45 smithi194 ceph-mon[32464]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:46.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.062+0000 7fc4270ab640 1 -- 172.21.15.119:0/630069478 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200fd6b0 msgr2=0x7fc4200fbae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:46.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.062+0000 7fc4270ab640 1 --2- 172.21.15.119:0/630069478 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200fd6b0 0x7fc4200fbae0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc40c004950 tx=0x7fc40c009da0 comp rx=0 tx=0).stop 2024-09-10T18:58:46.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.062+0000 7fc4270ab640 1 -- 172.21.15.119:0/630069478 shutdown_connections 2024-09-10T18:58:46.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.062+0000 7fc4270ab640 1 --2- 172.21.15.119:0/630069478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200fc210 0x7fc420103d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:46.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.062+0000 7fc4270ab640 1 --2- 172.21.15.119:0/630069478 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200fd6b0 0x7fc4200fbae0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:46.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.062+0000 7fc4270ab640 1 --2- 172.21.15.119:0/630069478 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4200fcd60 0x7fc4200fd170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:46.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.062+0000 7fc4270ab640 1 -- 172.21.15.119:0/630069478 >> 172.21.15.119:0/630069478 conn(0x7fc4200f66a0 msgr2=0x7fc4200f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:58:46.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.062+0000 7fc4270ab640 1 -- 172.21.15.119:0/630069478 shutdown_connections 2024-09-10T18:58:46.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.062+0000 7fc4270ab640 1 -- 172.21.15.119:0/630069478 wait complete. 2024-09-10T18:58:46.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc4270ab640 1 Processor -- start 2024-09-10T18:58:46.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc4270ab640 1 -- start start 2024-09-10T18:58:46.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc4270ab640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4200fc210 0x7fc42010ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:46.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc4270ab640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200fcd60 0x7fc42010d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:46.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc4270ab640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200fd6b0 0x7fc4201143e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:46.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc4270ab640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4201149b0 con 0x7fc4200fcd60 2024-09-10T18:58:46.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc4270ab640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc420114b20 con 0x7fc4200fc210 2024-09-10T18:58:46.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc4270ab640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc420114ce0 con 0x7fc4200fd6b0 2024-09-10T18:58:46.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc41ffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200fcd60 0x7fc42010d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:46.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc425648640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200fd6b0 0x7fc4201143e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:46.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc41ffff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200fcd60 0x7fc42010d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42698/0 (socket says 172.21.15.119:42698) 2024-09-10T18:58:46.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc41ffff640 1 -- 172.21.15.119:0/228176711 learned_addr learned my addr 172.21.15.119:0/228176711 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:58:46.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc424e47640 1 --2- 172.21.15.119:0/228176711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4200fc210 0x7fc42010ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:46.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc41ffff640 1 -- 172.21.15.119:0/228176711 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200fd6b0 msgr2=0x7fc4201143e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:46.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc41ffff640 1 --2- 172.21.15.119:0/228176711 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200fd6b0 0x7fc4201143e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:46.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc41ffff640 1 -- 172.21.15.119:0/228176711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4200fc210 msgr2=0x7fc42010ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:58:46.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc41ffff640 1 --2- 172.21.15.119:0/228176711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4200fc210 0x7fc42010ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:58:46.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc41ffff640 1 -- 172.21.15.119:0/228176711 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4180098b0 con 0x7fc4200fcd60 2024-09-10T18:58:46.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.063+0000 7fc424e47640 1 --2- 172.21.15.119:0/228176711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4200fc210 0x7fc42010ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T18:58:46.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.064+0000 7fc41ffff640 1 --2- 172.21.15.119:0/228176711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200fcd60 0x7fc42010d380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc40c02a560 tx=0x7fc40c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:58:46.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.064+0000 7fc41d7fa640 1 -- 172.21.15.119:0/228176711 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c02ac30 con 0x7fc4200fcd60 2024-09-10T18:58:46.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.064+0000 7fc4270ab640 1 -- 172.21.15.119:0/228176711 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40c005040 con 0x7fc4200fcd60 2024-09-10T18:58:46.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.064+0000 7fc41d7fa640 1 -- 172.21.15.119:0/228176711 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fc40c02ad90 con 0x7fc4200fcd60 2024-09-10T18:58:46.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.064+0000 7fc41d7fa640 1 -- 172.21.15.119:0/228176711 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c033690 con 0x7fc4200fcd60 2024-09-10T18:58:46.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.064+0000 7fc4270ab640 1 -- 172.21.15.119:0/228176711 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc420115240 con 0x7fc4200fcd60 2024-09-10T18:58:46.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.065+0000 7fc4270ab640 1 -- 172.21.15.119:0/228176711 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc420107ee0 con 0x7fc4200fcd60 2024-09-10T18:58:46.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.065+0000 7fc41d7fa640 1 -- 172.21.15.119:0/228176711 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 14) v1 ==== 90690+0+0 (secure 0 0 0) 0x7fc40c030020 con 0x7fc4200fcd60 2024-09-10T18:58:46.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.066+0000 7fc41d7fa640 1 --2- 172.21.15.119:0/228176711 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc41006ed80 0x7fc410071230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:58:46.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.066+0000 7fc41d7fa640 1 -- 172.21.15.119:0/228176711 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1537+0+0 (secure 0 0 0) 0x7fc40c0ac920 con 0x7fc4200fcd60 2024-09-10T18:58:46.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.066+0000 7fc424e47640 1 --2- 172.21.15.119:0/228176711 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc41006ed80 0x7fc410071230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:58:46.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.066+0000 7fc424e47640 1 --2- 172.21.15.119:0/228176711 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc41006ed80 0x7fc410071230 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc414009a80 tx=0x7fc414008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:58:46.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.070+0000 7fc41d7fa640 1 -- 172.21.15.119:0/228176711 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fc40c07aba0 con 0x7fc4200fcd60 2024-09-10T18:58:46.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:58:46.231+0000 7fc4270ab640 1 -- 172.21.15.119:0/228176711 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc420107860 con 0x7fc41006ed80 2024-09-10T18:58:46.945 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:46 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:58:46.945 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:46 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:58:46.945 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:46 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:46.945 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:46 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:58:46.945 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:46 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:58:46.945 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:46 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:47.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:46 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:58:47.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:46 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:58:47.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:46 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:47.876 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:47 smithi119 ceph-mon[29577]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:47.877 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:47 smithi119 ceph-mon[29577]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:58:47.877 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:47 smithi119 ceph-mon[40674]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:47.877 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:47 smithi119 ceph-mon[40674]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:58:48.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:47 smithi194 ceph-mon[32464]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:47 smithi194 ceph-mon[32464]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:58:48.915 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3043563319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdc97929-345b-440d-9d0f-c64e3cbe8110"}]: dispatch 2024-09-10T18:58:48.915 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3043563319' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cdc97929-345b-440d-9d0f-c64e3cbe8110"}]': finished 2024-09-10T18:58:48.915 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:48 smithi119 ceph-mon[29577]: osdmap e10: 2 total, 1 up, 2 in 2024-09-10T18:58:48.915 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:48 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:58:48.915 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3043563319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdc97929-345b-440d-9d0f-c64e3cbe8110"}]: dispatch 2024-09-10T18:58:48.915 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3043563319' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cdc97929-345b-440d-9d0f-c64e3cbe8110"}]': finished 2024-09-10T18:58:48.915 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:48 smithi119 ceph-mon[40674]: osdmap e10: 2 total, 1 up, 2 in 2024-09-10T18:58:48.915 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:48 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:58:49.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3043563319' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cdc97929-345b-440d-9d0f-c64e3cbe8110"}]: dispatch 2024-09-10T18:58:49.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3043563319' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cdc97929-345b-440d-9d0f-c64e3cbe8110"}]': finished 2024-09-10T18:58:49.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:48 smithi194 ceph-mon[32464]: osdmap e10: 2 total, 1 up, 2 in 2024-09-10T18:58:49.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:48 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:58:50.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:49 smithi194 ceph-mon[32464]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:50.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4279213199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:58:50.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:49 smithi119 ceph-mon[29577]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:50.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4279213199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:58:50.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:49 smithi119 ceph-mon[40674]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:50.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4279213199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:58:52.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:51 smithi194 ceph-mon[32464]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:52.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:51 smithi119 ceph-mon[29577]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:52.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:51 smithi119 ceph-mon[40674]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:53.714 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:53 smithi119 ceph-mon[29577]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:53.714 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:53 smithi119 ceph-mon[40674]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:54.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:53 smithi194 ceph-mon[32464]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:55.808 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:55 smithi119 ceph-mon[40674]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:55.809 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-10T18:58:55.809 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:55.809 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:55 smithi119 ceph-mon[40674]: Deploying daemon osd.1 on smithi119 2024-09-10T18:58:55.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:55 smithi119 ceph-mon[29577]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:55.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-10T18:58:55.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:55.809 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:55 smithi119 ceph-mon[29577]: Deploying daemon osd.1 on smithi119 2024-09-10T18:58:56.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:55 smithi194 ceph-mon[32464]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:56.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-10T18:58:56.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:58:56.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:55 smithi194 ceph-mon[32464]: Deploying daemon osd.1 on smithi119 2024-09-10T18:58:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:57 smithi119 ceph-mon[29577]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:57.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:57 smithi119 ceph-mon[40674]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:58:58.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:57 smithi194 ceph-mon[32464]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:00.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:58:59 smithi194 ceph-mon[32464]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:00.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:58:59 smithi119 ceph-mon[29577]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:00.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:58:59 smithi119 ceph-mon[40674]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:01.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:00 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:01.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:00 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:01.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:00 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:01.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:00 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:01.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:00 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:01.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:00 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:00 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:00 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:00 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:02.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:01 smithi194 ceph-mon[32464]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:02.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:01 smithi119 ceph-mon[29577]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:02.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:01 smithi119 ceph-mon[40674]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:02.993 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 1 on host 'smithi119' 2024-09-10T18:59:02.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:02.991+0000 7fc41d7fa640 1 -- 172.21.15.119:0/228176711 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc420107860 con 0x7fc41006ed80 2024-09-10T18:59:02.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:02.993+0000 7fc4270ab640 1 -- 172.21.15.119:0/228176711 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc41006ed80 msgr2=0x7fc410071230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:02.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:02.993+0000 7fc4270ab640 1 --2- 172.21.15.119:0/228176711 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc41006ed80 0x7fc410071230 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc414009a80 tx=0x7fc414008040 comp rx=0 tx=0).stop 2024-09-10T18:59:02.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:02.993+0000 7fc4270ab640 1 -- 172.21.15.119:0/228176711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200fcd60 msgr2=0x7fc42010d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:02.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:02.993+0000 7fc4270ab640 1 --2- 172.21.15.119:0/228176711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200fcd60 0x7fc42010d380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc40c02a560 tx=0x7fc40c009da0 comp rx=0 tx=0).stop 2024-09-10T18:59:02.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:02.993+0000 7fc4270ab640 1 -- 172.21.15.119:0/228176711 shutdown_connections 2024-09-10T18:59:02.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:02.993+0000 7fc4270ab640 1 --2- 172.21.15.119:0/228176711 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4200fd6b0 0x7fc4201143e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:02.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:02.993+0000 7fc4270ab640 1 --2- 172.21.15.119:0/228176711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4200fcd60 0x7fc42010d380 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:02.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:02.993+0000 7fc4270ab640 1 --2- 172.21.15.119:0/228176711 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc41006ed80 0x7fc410071230 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:02.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:02.993+0000 7fc4270ab640 1 --2- 172.21.15.119:0/228176711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4200fc210 0x7fc42010ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:02.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:02.993+0000 7fc4270ab640 1 -- 172.21.15.119:0/228176711 >> 172.21.15.119:0/228176711 conn(0x7fc4200f66a0 msgr2=0x7fc4200f7f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:02.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:02.993+0000 7fc4270ab640 1 -- 172.21.15.119:0/228176711 shutdown_connections 2024-09-10T18:59:02.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:02.993+0000 7fc4270ab640 1 -- 172.21.15.119:0/228176711 wait complete. 2024-09-10T18:59:03.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[29577]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[29577]: from='osd.1 v2:172.21.15.119:6805/4018071000' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:59:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[29577]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:59:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:03.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[40674]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:03.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:03.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:03.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:03.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:03.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:03.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[40674]: from='osd.1 v2:172.21.15.119:6805/4018071000' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:59:03.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[40674]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:59:03.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:03.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:03.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:03 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:03.636 DEBUG:teuthology.orchestra.run.smithi119:osd.1> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.1.service 2024-09-10T18:59:03.639 INFO:tasks.cephadm:Deploying osd.2 on smithi119 with /dev/vg_nvme/lv_2... 2024-09-10T18:59:03.639 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-10T18:59:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:03 smithi194 ceph-mon[32464]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:03 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:03 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:03 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:03 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:03 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:03 smithi194 ceph-mon[32464]: from='osd.1 v2:172.21.15.119:6805/4018071000' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:59:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:03 smithi194 ceph-mon[32464]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T18:59:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:03 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:03 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:03 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:04.787 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:05.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:04 smithi194 ceph-mon[32464]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T18:59:05.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:04 smithi194 ceph-mon[32464]: osdmap e11: 2 total, 1 up, 2 in 2024-09-10T18:59:05.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:59:05.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:04 smithi194 ceph-mon[32464]: from='osd.1 v2:172.21.15.119:6805/4018071000' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:59:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:04 smithi194 ceph-mon[32464]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:59:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:04 smithi194 ceph-mon[32464]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:05.035 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:04 smithi119 ceph-mon[29577]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T18:59:05.035 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:04 smithi119 ceph-mon[29577]: osdmap e11: 2 total, 1 up, 2 in 2024-09-10T18:59:05.035 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:59:05.035 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:04 smithi119 ceph-mon[29577]: from='osd.1 v2:172.21.15.119:6805/4018071000' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:59:05.036 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:04 smithi119 ceph-mon[29577]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:59:05.036 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:04 smithi119 ceph-mon[29577]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:05.036 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:04 smithi119 ceph-mon[40674]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T18:59:05.036 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:04 smithi119 ceph-mon[40674]: osdmap e11: 2 total, 1 up, 2 in 2024-09-10T18:59:05.036 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:59:05.036 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:04 smithi119 ceph-mon[40674]: from='osd.1 v2:172.21.15.119:6805/4018071000' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:59:05.036 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:04 smithi119 ceph-mon[40674]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:59:05.036 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:04 smithi119 ceph-mon[40674]: pgmap v54: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:05.036 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 18:59:04 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1[50994]: 2024-09-10T18:59:04.722+0000 7f757ad6b640 -1 osd.1 0 waiting for initial osdmap 2024-09-10T18:59:05.037 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 18:59:04 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1[50994]: 2024-09-10T18:59:04.725+0000 7f75753a1640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T18:59:05.995 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:05 smithi119 ceph-mon[29577]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-10T18:59:05.996 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:05 smithi119 ceph-mon[29577]: osdmap e12: 2 total, 1 up, 2 in 2024-09-10T18:59:05.996 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:05 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:59:05.996 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:05 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:59:05.996 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:05 smithi119 ceph-mon[40674]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-10T18:59:05.996 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:05 smithi119 ceph-mon[40674]: osdmap e12: 2 total, 1 up, 2 in 2024-09-10T18:59:05.996 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:05 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:59:05.996 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:05 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:59:06.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:05 smithi194 ceph-mon[32464]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-10T18:59:06.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:05 smithi194 ceph-mon[32464]: osdmap e12: 2 total, 1 up, 2 in 2024-09-10T18:59:06.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:05 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:59:06.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:05 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:59:07.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:06 smithi194 ceph-mon[32464]: purged_snaps scrub starts 2024-09-10T18:59:07.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:06 smithi194 ceph-mon[32464]: purged_snaps scrub ok 2024-09-10T18:59:07.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:06 smithi194 ceph-mon[32464]: osd.1 v2:172.21.15.119:6805/4018071000 boot 2024-09-10T18:59:07.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:06 smithi194 ceph-mon[32464]: osdmap e13: 2 total, 2 up, 2 in 2024-09-10T18:59:07.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:06 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:59:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:06 smithi194 ceph-mon[32464]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:07.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:06 smithi119 ceph-mon[29577]: purged_snaps scrub starts 2024-09-10T18:59:07.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:06 smithi119 ceph-mon[29577]: purged_snaps scrub ok 2024-09-10T18:59:07.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:06 smithi119 ceph-mon[29577]: osd.1 v2:172.21.15.119:6805/4018071000 boot 2024-09-10T18:59:07.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:06 smithi119 ceph-mon[29577]: osdmap e13: 2 total, 2 up, 2 in 2024-09-10T18:59:07.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:06 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:59:07.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:06 smithi119 ceph-mon[29577]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:07.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:06 smithi119 ceph-mon[40674]: purged_snaps scrub starts 2024-09-10T18:59:07.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:06 smithi119 ceph-mon[40674]: purged_snaps scrub ok 2024-09-10T18:59:07.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:06 smithi119 ceph-mon[40674]: osd.1 v2:172.21.15.119:6805/4018071000 boot 2024-09-10T18:59:07.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:06 smithi119 ceph-mon[40674]: osdmap e13: 2 total, 2 up, 2 in 2024-09-10T18:59:07.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:06 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T18:59:07.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:06 smithi119 ceph-mon[40674]: pgmap v57: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T18:59:07.673 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-10T18:59:07.749 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-10T18:59:07.762 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-09-10T18:59:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-09-10T18:59:07.763 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112882 s, 929 MB/s 2024-09-10T18:59:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-09-10T18:59:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:07 smithi194 ceph-mon[32464]: osdmap e14: 2 total, 2 up, 2 in 2024-09-10T18:59:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:07 smithi194 ceph-mon[32464]: Detected new or changed devices on smithi119 2024-09-10T18:59:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:07 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:07 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:07 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:07 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:07 smithi194 ceph-mon[32464]: Adjusting osd_memory_target on smithi119 to 8032M 2024-09-10T18:59:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:07 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:07 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:07 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:07 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[29577]: osdmap e14: 2 total, 2 up, 2 in 2024-09-10T18:59:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[29577]: Detected new or changed devices on smithi119 2024-09-10T18:59:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[29577]: Adjusting osd_memory_target on smithi119 to 8032M 2024-09-10T18:59:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:08.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[40674]: osdmap e14: 2 total, 2 up, 2 in 2024-09-10T18:59:08.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[40674]: Detected new or changed devices on smithi119 2024-09-10T18:59:08.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:08.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:08.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:08.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:08.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[40674]: Adjusting osd_memory_target on smithi119 to 8032M 2024-09-10T18:59:08.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:08.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:08.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:08.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:07 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:08.357 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi119:vg_nvme/lv_2 2024-09-10T18:59:08.600 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:08.976 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:08 smithi119 ceph-mon[29577]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:08.976 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:08 smithi119 ceph-mon[40674]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:09.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:08 smithi194 ceph-mon[32464]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.154+0000 7f838f257640 1 -- 172.21.15.119:0/186896336 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83880fc850 msgr2=0x7f83880fcc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.154+0000 7f838f257640 1 --2- 172.21.15.119:0/186896336 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83880fc850 0x7f83880fcc60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f837c004890 tx=0x7f837c009da0 comp rx=0 tx=0).stop 2024-09-10T18:59:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.154+0000 7f838f257640 1 -- 172.21.15.119:0/186896336 shutdown_connections 2024-09-10T18:59:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.154+0000 7f838f257640 1 --2- 172.21.15.119:0/186896336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83880fe4d0 0x7f8388104d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.154+0000 7f838f257640 1 --2- 172.21.15.119:0/186896336 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83880fdb20 0x7f83880fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:10.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.154+0000 7f838f257640 1 --2- 172.21.15.119:0/186896336 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83880fc850 0x7f83880fcc60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.154+0000 7f838f257640 1 -- 172.21.15.119:0/186896336 >> 172.21.15.119:0/186896336 conn(0x7f83880f8010 msgr2=0x7f83880fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.154+0000 7f838f257640 1 -- 172.21.15.119:0/186896336 shutdown_connections 2024-09-10T18:59:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.154+0000 7f838f257640 1 -- 172.21.15.119:0/186896336 wait complete. 2024-09-10T18:59:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838f257640 1 Processor -- start 2024-09-10T18:59:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838f257640 1 -- start start 2024-09-10T18:59:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838f257640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83880fc850 0x7f838810cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:59:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838f257640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83880fdb20 0x7f838810d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:59:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838f257640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83880fe4d0 0x7f8388114290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:59:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838f257640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8388114860 con 0x7f83880fc850 2024-09-10T18:59:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838f257640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83881149d0 con 0x7f83880fdb20 2024-09-10T18:59:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838f257640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8388114b90 con 0x7f83880fe4d0 2024-09-10T18:59:10.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838cff3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83880fc850 0x7f838810cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:59:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838d7f4640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83880fe4d0 0x7f8388114290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:59:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838cff3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83880fc850 0x7f838810cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37236/0 (socket says 172.21.15.119:37236) 2024-09-10T18:59:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838d7f4640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83880fe4d0 0x7f8388114290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:39424/0 (socket says 172.21.15.119:39424) 2024-09-10T18:59:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838cff3640 1 -- 172.21.15.119:0/3584452032 learned_addr learned my addr 172.21.15.119:0/3584452032 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:59:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f8387fff640 1 --2- 172.21.15.119:0/3584452032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83880fdb20 0x7f838810d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:59:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838cff3640 1 -- 172.21.15.119:0/3584452032 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83880fe4d0 msgr2=0x7f8388114290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838cff3640 1 --2- 172.21.15.119:0/3584452032 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83880fe4d0 0x7f8388114290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838cff3640 1 -- 172.21.15.119:0/3584452032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83880fdb20 msgr2=0x7f838810d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838cff3640 1 --2- 172.21.15.119:0/3584452032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83880fdb20 0x7f838810d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838cff3640 1 -- 172.21.15.119:0/3584452032 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83800098b0 con 0x7f83880fc850 2024-09-10T18:59:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.155+0000 7f838d7f4640 1 --2- 172.21.15.119:0/3584452032 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83880fe4d0 0x7f8388114290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T18:59:10.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.156+0000 7f838cff3640 1 --2- 172.21.15.119:0/3584452032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83880fc850 0x7f838810cd30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f837c004860 tx=0x7f837c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:59:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.156+0000 7f83857fa640 1 -- 172.21.15.119:0/3584452032 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c0090d0 con 0x7f83880fc850 2024-09-10T18:59:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.156+0000 7f838f257640 1 -- 172.21.15.119:0/3584452032 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837c005040 con 0x7f83880fc850 2024-09-10T18:59:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.156+0000 7f83857fa640 1 -- 172.21.15.119:0/3584452032 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f837c03c8f0 con 0x7f83880fc850 2024-09-10T18:59:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.156+0000 7f83857fa640 1 -- 172.21.15.119:0/3584452032 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c033630 con 0x7f83880fc850 2024-09-10T18:59:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.156+0000 7f838f257640 1 -- 172.21.15.119:0/3584452032 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8388115150 con 0x7f83880fc850 2024-09-10T18:59:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.157+0000 7f838f257640 1 -- 172.21.15.119:0/3584452032 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838804bf60 con 0x7f83880fc850 2024-09-10T18:59:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.157+0000 7f83857fa640 1 -- 172.21.15.119:0/3584452032 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 14) v1 ==== 90690+0+0 (secure 0 0 0) 0x7f837c030020 con 0x7f83880fc850 2024-09-10T18:59:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.158+0000 7f83857fa640 1 --2- 172.21.15.119:0/3584452032 >> v2:172.21.15.119:6800/1094129222 conn(0x7f837806ed50 0x7f8378071200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:59:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.158+0000 7f8387fff640 1 --2- 172.21.15.119:0/3584452032 >> v2:172.21.15.119:6800/1094129222 conn(0x7f837806ed50 0x7f8378071200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:59:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.158+0000 7f83857fa640 1 -- 172.21.15.119:0/3584452032 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 1829+0+0 (secure 0 0 0) 0x7f837c0acee0 con 0x7f83880fc850 2024-09-10T18:59:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.158+0000 7f8387fff640 1 --2- 172.21.15.119:0/3584452032 >> v2:172.21.15.119:6800/1094129222 conn(0x7f837806ed50 0x7f8378071200 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8374004ed0 tx=0x7f837400b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:59:10.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.162+0000 7f83857fa640 1 -- 172.21.15.119:0/3584452032 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f837c07b160 con 0x7f83880fc850 2024-09-10T18:59:10.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:10.326+0000 7f838f257640 1 -- 172.21.15.119:0/3584452032 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f8388107800 con 0x7f837806ed50 2024-09-10T18:59:11.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:11 smithi119 ceph-mon[40674]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:11.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:11 smithi119 ceph-mon[40674]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:59:11.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:11 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:59:11.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:11 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:59:11.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:11 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:11 smithi119 ceph-mon[29577]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:11 smithi119 ceph-mon[29577]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:59:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:11 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:59:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:11 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:59:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:11 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:11.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:11 smithi194 ceph-mon[32464]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:11.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:11 smithi194 ceph-mon[32464]: from='client.14265 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:59:11.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:11 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:59:11.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:11 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:59:11.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:11 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:13.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:13 smithi194 ceph-mon[32464]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:13.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1925281059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a53d3e5b-f03d-4082-a6d5-af183c1e8ca4"}]: dispatch 2024-09-10T18:59:13.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a53d3e5b-f03d-4082-a6d5-af183c1e8ca4"}]: dispatch 2024-09-10T18:59:13.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a53d3e5b-f03d-4082-a6d5-af183c1e8ca4"}]': finished 2024-09-10T18:59:13.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:13 smithi194 ceph-mon[32464]: osdmap e15: 3 total, 2 up, 3 in 2024-09-10T18:59:13.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:13 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:13.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4041909110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:59:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:13 smithi119 ceph-mon[29577]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1925281059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a53d3e5b-f03d-4082-a6d5-af183c1e8ca4"}]: dispatch 2024-09-10T18:59:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a53d3e5b-f03d-4082-a6d5-af183c1e8ca4"}]: dispatch 2024-09-10T18:59:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a53d3e5b-f03d-4082-a6d5-af183c1e8ca4"}]': finished 2024-09-10T18:59:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:13 smithi119 ceph-mon[29577]: osdmap e15: 3 total, 2 up, 3 in 2024-09-10T18:59:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:13 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:13.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4041909110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:59:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:13 smithi119 ceph-mon[40674]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1925281059' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a53d3e5b-f03d-4082-a6d5-af183c1e8ca4"}]: dispatch 2024-09-10T18:59:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a53d3e5b-f03d-4082-a6d5-af183c1e8ca4"}]: dispatch 2024-09-10T18:59:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a53d3e5b-f03d-4082-a6d5-af183c1e8ca4"}]': finished 2024-09-10T18:59:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:13 smithi119 ceph-mon[40674]: osdmap e15: 3 total, 2 up, 3 in 2024-09-10T18:59:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:13 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4041909110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:59:15.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:15 smithi194 ceph-mon[32464]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:15.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:15 smithi119 ceph-mon[29577]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:15 smithi119 ceph-mon[40674]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:17.268 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:17 smithi119 ceph-mon[29577]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:17.269 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:17 smithi119 ceph-mon[40674]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:17.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:17 smithi194 ceph-mon[32464]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:19 smithi119 ceph-mon[40674]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-10T18:59:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:19 smithi119 ceph-mon[40674]: Deploying daemon osd.2 on smithi119 2024-09-10T18:59:19.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:19 smithi119 ceph-mon[29577]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:19.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-10T18:59:19.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:19.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:19 smithi119 ceph-mon[29577]: Deploying daemon osd.2 on smithi119 2024-09-10T18:59:19.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:19 smithi194 ceph-mon[32464]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:19.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-10T18:59:19.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:19.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:19 smithi194 ceph-mon[32464]: Deploying daemon osd.2 on smithi119 2024-09-10T18:59:21.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:21 smithi119 ceph-mon[29577]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:21.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:21 smithi119 ceph-mon[40674]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:21.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:21 smithi194 ceph-mon[32464]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:23.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:23 smithi194 ceph-mon[32464]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:23.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:23 smithi119 ceph-mon[29577]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:23.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:23 smithi119 ceph-mon[40674]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:24.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:24 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:24.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:24 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:24.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:24 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:24.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:24 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:24.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:24 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:24.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:24 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:24.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:24 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:24.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:24 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:24.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:24 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:25.552 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:25 smithi119 ceph-mon[29577]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:25.553 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:25 smithi119 ceph-mon[40674]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:25.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:25 smithi194 ceph-mon[32464]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:26.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:26.898+0000 7f83857fa640 1 -- 172.21.15.119:0/3584452032 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8388107800 con 0x7f837806ed50 2024-09-10T18:59:26.900 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 2 on host 'smithi119' 2024-09-10T18:59:26.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:26.900+0000 7f838f257640 1 -- 172.21.15.119:0/3584452032 >> v2:172.21.15.119:6800/1094129222 conn(0x7f837806ed50 msgr2=0x7f8378071200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:26.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:26.900+0000 7f838f257640 1 --2- 172.21.15.119:0/3584452032 >> v2:172.21.15.119:6800/1094129222 conn(0x7f837806ed50 0x7f8378071200 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8374004ed0 tx=0x7f837400b040 comp rx=0 tx=0).stop 2024-09-10T18:59:26.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:26.900+0000 7f838f257640 1 -- 172.21.15.119:0/3584452032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83880fc850 msgr2=0x7f838810cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:26.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:26.900+0000 7f838f257640 1 --2- 172.21.15.119:0/3584452032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83880fc850 0x7f838810cd30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f837c004860 tx=0x7f837c009da0 comp rx=0 tx=0).stop 2024-09-10T18:59:26.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:26.900+0000 7f838f257640 1 -- 172.21.15.119:0/3584452032 shutdown_connections 2024-09-10T18:59:26.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:26.900+0000 7f838f257640 1 --2- 172.21.15.119:0/3584452032 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83880fe4d0 0x7f8388114290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:26.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:26.900+0000 7f838f257640 1 --2- 172.21.15.119:0/3584452032 >> v2:172.21.15.119:6800/1094129222 conn(0x7f837806ed50 0x7f8378071200 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:26.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:26.900+0000 7f838f257640 1 --2- 172.21.15.119:0/3584452032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83880fdb20 0x7f838810d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:26.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:26.900+0000 7f838f257640 1 --2- 172.21.15.119:0/3584452032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83880fc850 0x7f838810cd30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:26.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:26.900+0000 7f838f257640 1 -- 172.21.15.119:0/3584452032 >> 172.21.15.119:0/3584452032 conn(0x7f83880f8010 msgr2=0x7f83880fb360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:26.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:26.900+0000 7f838f257640 1 -- 172.21.15.119:0/3584452032 shutdown_connections 2024-09-10T18:59:26.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:26.900+0000 7f838f257640 1 -- 172.21.15.119:0/3584452032 wait complete. 2024-09-10T18:59:27.542 DEBUG:teuthology.orchestra.run.smithi119:osd.2> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.2.service 2024-09-10T18:59:27.546 INFO:tasks.cephadm:Deploying osd.3 on smithi119 with /dev/vg_nvme/lv_1... 2024-09-10T18:59:27.546 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-10T18:59:27.762 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[29577]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:27.763 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:27.763 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:27.763 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:27.763 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:27.763 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[29577]: from='osd.2 v2:172.21.15.119:6809/150641920' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T18:59:27.763 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:27.763 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:27.763 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:27.763 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:27.763 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[40674]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:27.763 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:27.763 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:27.764 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:27.764 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:27.764 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[40674]: from='osd.2 v2:172.21.15.119:6809/150641920' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T18:59:27.764 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:27.764 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:27.764 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:27.764 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:27 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:27.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:27 smithi194 ceph-mon[32464]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:27.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:27 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:27.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:27 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:27.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:27 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:27.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:27 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:27 smithi194 ceph-mon[32464]: from='osd.2 v2:172.21.15.119:6809/150641920' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T18:59:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:27 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:27 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:27 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:27 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:28.289 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:28.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:28 smithi119 ceph-mon[40674]: from='osd.2 v2:172.21.15.119:6809/150641920' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-10T18:59:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:28 smithi119 ceph-mon[40674]: osdmap e16: 3 total, 2 up, 3 in 2024-09-10T18:59:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:28 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:28 smithi119 ceph-mon[40674]: from='osd.2 v2:172.21.15.119:6809/150641920' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:59:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:28 smithi119 ceph-mon[29577]: from='osd.2 v2:172.21.15.119:6809/150641920' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-10T18:59:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:28 smithi119 ceph-mon[29577]: osdmap e16: 3 total, 2 up, 3 in 2024-09-10T18:59:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:28 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:28 smithi119 ceph-mon[29577]: from='osd.2 v2:172.21.15.119:6809/150641920' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:59:28.818 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 18:59:28 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2[56942]: 2024-09-10T18:59:28.519+0000 7f6955063640 -1 osd.2 0 waiting for initial osdmap 2024-09-10T18:59:28.818 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 18:59:28 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2[56942]: 2024-09-10T18:59:28.523+0000 7f694de96640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T18:59:29.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:28 smithi194 ceph-mon[32464]: from='osd.2 v2:172.21.15.119:6809/150641920' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-10T18:59:29.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:28 smithi194 ceph-mon[32464]: osdmap e16: 3 total, 2 up, 3 in 2024-09-10T18:59:29.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:28 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:29.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:28 smithi194 ceph-mon[32464]: from='osd.2 v2:172.21.15.119:6809/150641920' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:59:29.776 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:29 smithi119 ceph-mon[29577]: purged_snaps scrub starts 2024-09-10T18:59:29.776 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:29 smithi119 ceph-mon[29577]: purged_snaps scrub ok 2024-09-10T18:59:29.776 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:29 smithi119 ceph-mon[29577]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:29.776 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:29 smithi119 ceph-mon[29577]: from='osd.2 v2:172.21.15.119:6809/150641920' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-10T18:59:29.776 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:29 smithi119 ceph-mon[29577]: osdmap e17: 3 total, 2 up, 3 in 2024-09-10T18:59:29.776 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:29 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:29.776 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:29 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:29.776 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:29 smithi119 ceph-mon[40674]: purged_snaps scrub starts 2024-09-10T18:59:29.776 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:29 smithi119 ceph-mon[40674]: purged_snaps scrub ok 2024-09-10T18:59:29.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:29 smithi119 ceph-mon[40674]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:29.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:29 smithi119 ceph-mon[40674]: from='osd.2 v2:172.21.15.119:6809/150641920' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-10T18:59:29.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:29 smithi119 ceph-mon[40674]: osdmap e17: 3 total, 2 up, 3 in 2024-09-10T18:59:29.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:29 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:29.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:29 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:30.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:29 smithi194 ceph-mon[32464]: purged_snaps scrub starts 2024-09-10T18:59:30.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:29 smithi194 ceph-mon[32464]: purged_snaps scrub ok 2024-09-10T18:59:30.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:29 smithi194 ceph-mon[32464]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:30.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:29 smithi194 ceph-mon[32464]: from='osd.2 v2:172.21.15.119:6809/150641920' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-10T18:59:30.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:29 smithi194 ceph-mon[32464]: osdmap e17: 3 total, 2 up, 3 in 2024-09-10T18:59:30.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:29 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:30.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:29 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:30.647 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:30 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:30.647 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:30 smithi119 ceph-mon[29577]: osd.2 v2:172.21.15.119:6809/150641920 boot 2024-09-10T18:59:30.647 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:30 smithi119 ceph-mon[29577]: osdmap e18: 3 total, 3 up, 3 in 2024-09-10T18:59:30.647 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:30 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:30.648 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:30 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-10T18:59:30.648 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:30 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:30.648 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:30 smithi119 ceph-mon[40674]: osd.2 v2:172.21.15.119:6809/150641920 boot 2024-09-10T18:59:30.648 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:30 smithi119 ceph-mon[40674]: osdmap e18: 3 total, 3 up, 3 in 2024-09-10T18:59:30.648 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:30 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:30.648 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:30 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-10T18:59:31.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:30 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:31.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:30 smithi194 ceph-mon[32464]: osd.2 v2:172.21.15.119:6809/150641920 boot 2024-09-10T18:59:31.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:30 smithi194 ceph-mon[32464]: osdmap e18: 3 total, 3 up, 3 in 2024-09-10T18:59:31.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:30 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T18:59:31.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:30 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-10T18:59:31.191 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-10T18:59:31.283 INFO:teuthology.orchestra.run.smithi119.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-10T18:59:31.297 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10+0 records in 2024-09-10T18:59:31.297 INFO:teuthology.orchestra.run.smithi119.stderr:10+0 records out 2024-09-10T18:59:31.297 INFO:teuthology.orchestra.run.smithi119.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123748 s, 847 MB/s 2024-09-10T18:59:31.297 INFO:teuthology.orchestra.run.smithi119.stderr:--> Zapping successful for: 2024-09-10T18:59:31.815 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[29577]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:31.815 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-10T18:59:31.815 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[29577]: osdmap e19: 3 total, 3 up, 3 in 2024-09-10T18:59:31.815 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:31.815 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:31.815 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:31.815 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:31.815 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:31.815 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:31.815 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:31.815 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:31.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:31.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:31.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[40674]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:31.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-10T18:59:31.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[40674]: osdmap e19: 3 total, 3 up, 3 in 2024-09-10T18:59:31.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:31.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:31.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:31.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:31.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:31.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:31 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:32.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:31 smithi194 ceph-mon[32464]: pgmap v74: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T18:59:32.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:31 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-10T18:59:32.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:31 smithi194 ceph-mon[32464]: osdmap e19: 3 total, 3 up, 3 in 2024-09-10T18:59:32.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:31 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T18:59:32.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:31 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:32.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:31 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:32.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:31 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:32.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:31 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:32.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:31 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:32.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:31 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:32.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:31 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:32.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:31 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:32.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:31 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:32.066 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 18:59:31 smithi119 sudo[61035]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T18:59:32.067 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 18:59:31 smithi119 sudo[61035]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:59:32.067 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 18:59:31 smithi119 sudo[61035]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:59:32.067 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 18:59:31 smithi119 sudo[61035]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:59:32.080 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi119:vg_nvme/lv_1 2024-09-10T18:59:32.331 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T18:59:32.402 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61056]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T18:59:32.403 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61056]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:59:32.403 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61056]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:59:32.403 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61056]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:59:32.666 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61114]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T18:59:32.666 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61114]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:59:32.666 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61114]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:59:32.666 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61114]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:59:32.667 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:32 smithi119 ceph-mon[29577]: Detected new or changed devices on smithi119 2024-09-10T18:59:32.667 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:32 smithi119 ceph-mon[29577]: Adjusting osd_memory_target on smithi119 to 5354M 2024-09-10T18:59:32.667 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:32 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:32.667 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:32 smithi119 ceph-mon[29577]: osdmap e20: 3 total, 3 up, 3 in 2024-09-10T18:59:32.667 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:32 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:32.667 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:32 smithi119 ceph-mon[40674]: Detected new or changed devices on smithi119 2024-09-10T18:59:32.667 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:32 smithi119 ceph-mon[40674]: Adjusting osd_memory_target on smithi119 to 5354M 2024-09-10T18:59:32.667 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:32 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:32.667 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:32 smithi119 ceph-mon[40674]: osdmap e20: 3 total, 3 up, 3 in 2024-09-10T18:59:32.667 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:32 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:32.917 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61137]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T18:59:32.917 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61137]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:59:32.917 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61137]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:59:32.917 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61137]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:59:33.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:32 smithi194 ceph-mon[32464]: Detected new or changed devices on smithi119 2024-09-10T18:59:33.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:32 smithi194 ceph-mon[32464]: Adjusting osd_memory_target on smithi119 to 5354M 2024-09-10T18:59:33.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:32 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-10T18:59:33.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:32 smithi194 ceph-mon[32464]: osdmap e20: 3 total, 3 up, 3 in 2024-09-10T18:59:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:32 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:33.172 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61159]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T18:59:33.172 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61159]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:59:33.172 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61159]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:59:33.172 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 18:59:32 smithi119 sudo[61159]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:59:33.427 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 18:59:33 smithi119 sudo[61170]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T18:59:33.427 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 18:59:33 smithi119 sudo[61170]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:59:33.427 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 18:59:33 smithi119 sudo[61170]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:59:33.427 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 18:59:33 smithi119 sudo[61170]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:59:33.689 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:33 smithi119 sudo[61173]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T18:59:33.689 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:33 smithi119 sudo[61173]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:59:33.689 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:33 smithi119 sudo[61173]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:59:33.689 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:33 smithi119 sudo[61173]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:59:33.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 -- 172.21.15.119:0/1167047730 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b080fdb30 msgr2=0x7f8b080fdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:33.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 --2- 172.21.15.119:0/1167047730 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b080fdb30 0x7f8b080fdfa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8af0004890 tx=0x7f8af0009da0 comp rx=0 tx=0).stop 2024-09-10T18:59:33.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 -- 172.21.15.119:0/1167047730 shutdown_connections 2024-09-10T18:59:33.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 --2- 172.21.15.119:0/1167047730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b080fe4e0 0x7f8b08104d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:33.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 --2- 172.21.15.119:0/1167047730 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b080fdb30 0x7f8b080fdfa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:33.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 --2- 172.21.15.119:0/1167047730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b080fc860 0x7f8b080fcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:33.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 -- 172.21.15.119:0/1167047730 >> 172.21.15.119:0/1167047730 conn(0x7f8b080f8040 msgr2=0x7f8b080fa450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:33.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 -- 172.21.15.119:0/1167047730 shutdown_connections 2024-09-10T18:59:33.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 -- 172.21.15.119:0/1167047730 wait complete. 2024-09-10T18:59:33.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 Processor -- start 2024-09-10T18:59:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 -- start start 2024-09-10T18:59:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b080fc860 0x7f8b0810cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:59:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b080fdb30 0x7f8b0810d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:59:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b080fe4e0 0x7f8b081142b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:59:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b08114880 con 0x7f8b080fe4e0 2024-09-10T18:59:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b081149f0 con 0x7f8b080fc860 2024-09-10T18:59:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.904+0000 7f8b0cb43640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b08114bb0 con 0x7f8b080fdb30 2024-09-10T18:59:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8afdd9b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b080fdb30 0x7f8b0810d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:59:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8b06d9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b080fe4e0 0x7f8b081142b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:59:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8afdd9b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b080fdb30 0x7f8b0810d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:33712/0 (socket says 172.21.15.119:33712) 2024-09-10T18:59:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8b06d9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b080fe4e0 0x7f8b081142b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60774/0 (socket says 172.21.15.119:60774) 2024-09-10T18:59:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8afdd9b640 1 -- 172.21.15.119:0/3815683702 learned_addr learned my addr 172.21.15.119:0/3815683702 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T18:59:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8b0659c640 1 --2- 172.21.15.119:0/3815683702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b080fc860 0x7f8b0810cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:59:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8afdd9b640 1 -- 172.21.15.119:0/3815683702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b080fc860 msgr2=0x7f8b0810cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8afdd9b640 1 --2- 172.21.15.119:0/3815683702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b080fc860 0x7f8b0810cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8afdd9b640 1 -- 172.21.15.119:0/3815683702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b080fe4e0 msgr2=0x7f8b081142b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8afdd9b640 1 --2- 172.21.15.119:0/3815683702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b080fe4e0 0x7f8b081142b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8afdd9b640 1 -- 172.21.15.119:0/3815683702 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b000098b0 con 0x7f8b080fdb30 2024-09-10T18:59:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8b06d9d640 1 --2- 172.21.15.119:0/3815683702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b080fe4e0 0x7f8b081142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T18:59:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8afdd9b640 1 --2- 172.21.15.119:0/3815683702 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b080fdb30 0x7f8b0810d290 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8af0004860 tx=0x7f8af0009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:59:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8aff7fe640 1 -- 172.21.15.119:0/3815683702 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af00090d0 con 0x7f8b080fdb30 2024-09-10T18:59:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8aff7fe640 1 -- 172.21.15.119:0/3815683702 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f8af003c8f0 con 0x7f8b080fdb30 2024-09-10T18:59:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8b0cb43640 1 -- 172.21.15.119:0/3815683702 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af0005040 con 0x7f8b080fdb30 2024-09-10T18:59:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8aff7fe640 1 -- 172.21.15.119:0/3815683702 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8af0033630 con 0x7f8b080fdb30 2024-09-10T18:59:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.905+0000 7f8b0cb43640 1 -- 172.21.15.119:0/3815683702 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b08115170 con 0x7f8b080fdb30 2024-09-10T18:59:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.906+0000 7f8b0cb43640 1 -- 172.21.15.119:0/3815683702 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b0804bf60 con 0x7f8b080fdb30 2024-09-10T18:59:33.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.907+0000 7f8aff7fe640 1 -- 172.21.15.119:0/3815683702 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 15) v1 ==== 90746+0+0 (secure 0 0 0) 0x7f8af0030020 con 0x7f8b080fdb30 2024-09-10T18:59:33.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.907+0000 7f8aff7fe640 1 --2- 172.21.15.119:0/3815683702 >> v2:172.21.15.119:6800/1094129222 conn(0x7f8af406ede0 0x7f8af4071290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:59:33.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.907+0000 7f8aff7fe640 1 -- 172.21.15.119:0/3815683702 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 2507+0+0 (secure 0 0 0) 0x7f8af00ad2f0 con 0x7f8b080fdb30 2024-09-10T18:59:33.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.907+0000 7f8b0659c640 1 --2- 172.21.15.119:0/3815683702 >> v2:172.21.15.119:6800/1094129222 conn(0x7f8af406ede0 0x7f8af4071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:59:33.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.908+0000 7f8b0659c640 1 --2- 172.21.15.119:0/3815683702 >> v2:172.21.15.119:6800/1094129222 conn(0x7f8af406ede0 0x7f8af4071290 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8b0810e210 tx=0x7f8af800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:59:33.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:33.911+0000 7f8aff7fe640 1 -- 172.21.15.119:0/3815683702 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f8af007b050 con 0x7f8b080fdb30 2024-09-10T18:59:33.958 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:33 smithi194 ceph-mon[32464]: pgmap v77: 1 pgs: 1 active+clean; 320 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:33.958 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:33 smithi194 ceph-mon[32464]: osdmap e21: 3 total, 3 up, 3 in 2024-09-10T18:59:33.958 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:33 smithi194 ceph-mon[32464]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-10T18:59:33.958 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:33 smithi194 ceph-mon[32464]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:59:33.958 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:33 smithi194 sudo[35235]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T18:59:33.958 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:33 smithi194 sudo[35235]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:59:33.958 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:33 smithi194 sudo[35235]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:59:33.958 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:33 smithi194 sudo[35235]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:59:34.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:33 smithi119 ceph-mon[40674]: pgmap v77: 1 pgs: 1 active+clean; 320 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:34.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:33 smithi119 ceph-mon[40674]: osdmap e21: 3 total, 3 up, 3 in 2024-09-10T18:59:34.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:33 smithi119 ceph-mon[40674]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-10T18:59:34.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:33 smithi119 ceph-mon[40674]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:59:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:33 smithi119 sudo[61196]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T18:59:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:33 smithi119 sudo[61196]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:59:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:33 smithi119 sudo[61196]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:59:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:33 smithi119 sudo[61196]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:59:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:33 smithi119 ceph-mon[29577]: pgmap v77: 1 pgs: 1 active+clean; 320 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:33 smithi119 ceph-mon[29577]: osdmap e21: 3 total, 3 up, 3 in 2024-09-10T18:59:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:33 smithi119 ceph-mon[29577]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-10T18:59:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:33 smithi119 ceph-mon[29577]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:59:34.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:34.075+0000 7f8b0cb43640 1 -- 172.21.15.119:0/3815683702 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8b08107800 con 0x7f8af406ede0 2024-09-10T18:59:34.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 sudo[35238]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T18:59:34.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 sudo[35238]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:59:34.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 sudo[35238]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:59:34.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 sudo[35238]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:59:34.728 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 sudo[61366]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T18:59:34.728 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 sudo[61366]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:59:34.728 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 sudo[61366]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:59:34.728 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 sudo[61366]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:59:35.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:59:35.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:59:35.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:59:35.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:59:35.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:59:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:59:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:59:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:59:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:59:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:59:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:59:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:59:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:59:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:34 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:59:35.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:59:35.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:59:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:59:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:59:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:59:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:59:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:59:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:59:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:59:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:59:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:59:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:59:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:59:35.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:59:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 sudo[61389]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T18:59:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 sudo[61389]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T18:59:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 sudo[61389]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T18:59:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 sudo[61389]: pam_unix(sudo:session): session closed for user root 2024-09-10T18:59:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:59:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:59:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:59:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:59:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:59:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:59:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:59:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:59:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:59:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:59:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:59:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T18:59:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T18:59:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:34 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T18:59:35.960 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:35 smithi119 ceph-mon[29577]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:59:35.960 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:35 smithi119 ceph-mon[29577]: pgmap v79: 1 pgs: 1 active+clean; 320 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:35.960 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:35 smithi119 ceph-mon[29577]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:59:35.960 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:35 smithi119 ceph-mon[29577]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:59:35.960 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:35 smithi119 ceph-mon[40674]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:59:35.960 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:35 smithi119 ceph-mon[40674]: pgmap v79: 1 pgs: 1 active+clean; 320 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:35.960 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:35 smithi119 ceph-mon[40674]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:59:35.960 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:35 smithi119 ceph-mon[40674]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:59:36.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:35 smithi194 ceph-mon[32464]: from='client.24203 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi119:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:59:36.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:35 smithi194 ceph-mon[32464]: pgmap v79: 1 pgs: 1 active+clean; 320 KiB data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:36.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:35 smithi194 ceph-mon[32464]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T18:59:36.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:35 smithi194 ceph-mon[32464]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T18:59:36.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:36 smithi119 ceph-mon[29577]: pgmap v80: 1 pgs: 1 active+clean; 320 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:36.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1590881564' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "291e81da-6fc3-44a4-880c-dd6c0ad00c13"}]: dispatch 2024-09-10T18:59:36.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1590881564' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "291e81da-6fc3-44a4-880c-dd6c0ad00c13"}]': finished 2024-09-10T18:59:36.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:36 smithi119 ceph-mon[29577]: osdmap e22: 4 total, 3 up, 4 in 2024-09-10T18:59:36.947 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:36 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:36.948 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:36 smithi119 ceph-mon[40674]: pgmap v80: 1 pgs: 1 active+clean; 320 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:36.948 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1590881564' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "291e81da-6fc3-44a4-880c-dd6c0ad00c13"}]: dispatch 2024-09-10T18:59:36.948 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1590881564' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "291e81da-6fc3-44a4-880c-dd6c0ad00c13"}]': finished 2024-09-10T18:59:36.948 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:36 smithi119 ceph-mon[40674]: osdmap e22: 4 total, 3 up, 4 in 2024-09-10T18:59:36.948 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:36 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:37.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:36 smithi194 ceph-mon[32464]: pgmap v80: 1 pgs: 1 active+clean; 320 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:37.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1590881564' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "291e81da-6fc3-44a4-880c-dd6c0ad00c13"}]: dispatch 2024-09-10T18:59:37.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1590881564' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "291e81da-6fc3-44a4-880c-dd6c0ad00c13"}]': finished 2024-09-10T18:59:37.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:36 smithi194 ceph-mon[32464]: osdmap e22: 4 total, 3 up, 4 in 2024-09-10T18:59:37.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:36 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:38.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2776406566' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:59:38.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2776406566' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:59:38.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2776406566' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:59:39.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:38 smithi194 ceph-mon[32464]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-09-10T18:59:39.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:38 smithi119 ceph-mon[29577]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-09-10T18:59:39.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:38 smithi119 ceph-mon[40674]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 4.5 KiB/s rd, 218 KiB/s wr, 14 op/s 2024-09-10T18:59:41.288 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:41 smithi119 ceph-mon[29577]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-09-10T18:59:41.288 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:41 smithi119 ceph-mon[40674]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-09-10T18:59:41.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:41 smithi194 ceph-mon[32464]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 3.4 KiB/s rd, 163 KiB/s wr, 11 op/s 2024-09-10T18:59:43.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:43 smithi194 ceph-mon[32464]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.8 KiB/s rd, 136 KiB/s wr, 9 op/s 2024-09-10T18:59:43.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:43 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-10T18:59:43.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:43 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:43.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:43 smithi194 ceph-mon[32464]: Deploying daemon osd.3 on smithi119 2024-09-10T18:59:43.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:43 smithi119 ceph-mon[40674]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.8 KiB/s rd, 136 KiB/s wr, 9 op/s 2024-09-10T18:59:43.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:43 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-10T18:59:43.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:43 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:43.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:43 smithi119 ceph-mon[40674]: Deploying daemon osd.3 on smithi119 2024-09-10T18:59:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:43 smithi119 ceph-mon[29577]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.8 KiB/s rd, 136 KiB/s wr, 9 op/s 2024-09-10T18:59:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-10T18:59:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:43 smithi119 ceph-mon[29577]: Deploying daemon osd.3 on smithi119 2024-09-10T18:59:45.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:45 smithi194 ceph-mon[32464]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-09-10T18:59:45.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:45 smithi119 ceph-mon[29577]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-09-10T18:59:45.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:45 smithi119 ceph-mon[40674]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-09-10T18:59:47.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:47 smithi119 ceph-mon[29577]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-09-10T18:59:47.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:47 smithi119 ceph-mon[40674]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-09-10T18:59:47.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:47 smithi194 ceph-mon[32464]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.7 KiB/s rd, 131 KiB/s wr, 8 op/s 2024-09-10T18:59:48.315 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:48 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:48.315 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:48 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:48.315 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:48 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:48.315 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:48 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:48.315 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:48 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:48.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:48 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:48.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:48 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:48.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:48 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:48.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:48 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:49.647 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:49 smithi119 ceph-mon[29577]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.3 KiB/s rd, 111 KiB/s wr, 7 op/s 2024-09-10T18:59:49.647 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:49 smithi119 ceph-mon[40674]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.3 KiB/s rd, 111 KiB/s wr, 7 op/s 2024-09-10T18:59:50.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:49 smithi194 ceph-mon[32464]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail; 2.3 KiB/s rd, 111 KiB/s wr, 7 op/s 2024-09-10T18:59:50.644 INFO:teuthology.orchestra.run.smithi119.stdout:Created osd(s) 3 on host 'smithi119' 2024-09-10T18:59:50.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:50.642+0000 7f8aff7fe640 1 -- 172.21.15.119:0/3815683702 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8b08107800 con 0x7f8af406ede0 2024-09-10T18:59:50.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:50.644+0000 7f8b0cb43640 1 -- 172.21.15.119:0/3815683702 >> v2:172.21.15.119:6800/1094129222 conn(0x7f8af406ede0 msgr2=0x7f8af4071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:50.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:50.644+0000 7f8b0cb43640 1 --2- 172.21.15.119:0/3815683702 >> v2:172.21.15.119:6800/1094129222 conn(0x7f8af406ede0 0x7f8af4071290 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8b0810e210 tx=0x7f8af800b040 comp rx=0 tx=0).stop 2024-09-10T18:59:50.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:50.644+0000 7f8b0cb43640 1 -- 172.21.15.119:0/3815683702 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b080fdb30 msgr2=0x7f8b0810d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:50.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:50.644+0000 7f8b0cb43640 1 --2- 172.21.15.119:0/3815683702 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b080fdb30 0x7f8b0810d290 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8af0004860 tx=0x7f8af0009da0 comp rx=0 tx=0).stop 2024-09-10T18:59:50.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:50.644+0000 7f8b0cb43640 1 -- 172.21.15.119:0/3815683702 shutdown_connections 2024-09-10T18:59:50.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:50.644+0000 7f8b0cb43640 1 --2- 172.21.15.119:0/3815683702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b080fe4e0 0x7f8b081142b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:50.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:50.644+0000 7f8b0cb43640 1 --2- 172.21.15.119:0/3815683702 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8b080fdb30 0x7f8b0810d290 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:50.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:50.644+0000 7f8b0cb43640 1 --2- 172.21.15.119:0/3815683702 >> v2:172.21.15.119:6800/1094129222 conn(0x7f8af406ede0 0x7f8af4071290 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:50.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:50.644+0000 7f8b0cb43640 1 --2- 172.21.15.119:0/3815683702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b080fc860 0x7f8b0810cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:50.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:50.644+0000 7f8b0cb43640 1 -- 172.21.15.119:0/3815683702 >> 172.21.15.119:0/3815683702 conn(0x7f8b080f8040 msgr2=0x7f8b080fb370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:50.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:50.644+0000 7f8b0cb43640 1 -- 172.21.15.119:0/3815683702 shutdown_connections 2024-09-10T18:59:50.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T18:59:50.644+0000 7f8b0cb43640 1 -- 172.21.15.119:0/3815683702 wait complete. 2024-09-10T18:59:51.308 DEBUG:teuthology.orchestra.run.smithi119:osd.3> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.3.service 2024-09-10T18:59:51.310 INFO:tasks.cephadm:Deploying osd.4 on smithi194 with /dev/vg_nvme/lv_4... 2024-09-10T18:59:51.310 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-10T18:59:51.562 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:59:51.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:51 smithi194 ceph-mon[32464]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:51.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:51 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:51.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:51 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:51.892 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:51 smithi194 ceph-mon[32464]: from='osd.3 v2:172.21.15.119:6813/783850966' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:59:52.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:51 smithi119 ceph-mon[29577]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:52.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:51 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:52.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:51 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:52.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:51 smithi119 ceph-mon[29577]: from='osd.3 v2:172.21.15.119:6813/783850966' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:59:52.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:51 smithi119 ceph-mon[40674]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:52.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:51 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:52.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:51 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:52.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:51 smithi119 ceph-mon[40674]: from='osd.3 v2:172.21.15.119:6813/783850966' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T18:59:52.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:52 smithi119 ceph-mon[40674]: from='osd.3 v2:172.21.15.119:6813/783850966' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-10T18:59:52.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:52 smithi119 ceph-mon[40674]: osdmap e23: 4 total, 3 up, 4 in 2024-09-10T18:59:52.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:52 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:52.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:52 smithi119 ceph-mon[40674]: from='osd.3 v2:172.21.15.119:6813/783850966' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:59:52.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:52 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:52.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:52 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:52.871 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:52 smithi119 ceph-mon[29577]: from='osd.3 v2:172.21.15.119:6813/783850966' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-10T18:59:52.871 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:52 smithi119 ceph-mon[29577]: osdmap e23: 4 total, 3 up, 4 in 2024-09-10T18:59:52.871 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:52 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:52.871 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:52 smithi119 ceph-mon[29577]: from='osd.3 v2:172.21.15.119:6813/783850966' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:59:52.871 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:52 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:52.871 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:52 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:52.871 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 18:59:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3[63296]: 2024-09-10T18:59:52.657+0000 7fb6b95dc640 -1 osd.3 0 waiting for initial osdmap 2024-09-10T18:59:52.871 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 18:59:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3[63296]: 2024-09-10T18:59:52.663+0000 7fb6b5415640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T18:59:52.967 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:52 smithi194 ceph-mon[32464]: from='osd.3 v2:172.21.15.119:6813/783850966' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-10T18:59:52.967 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:52 smithi194 ceph-mon[32464]: osdmap e23: 4 total, 3 up, 4 in 2024-09-10T18:59:52.967 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:52.967 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:52 smithi194 ceph-mon[32464]: from='osd.3 v2:172.21.15.119:6813/783850966' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]: dispatch 2024-09-10T18:59:52.967 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:52.967 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:53.199 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-10T18:59:53.279 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-10T18:59:53.293 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-09-10T18:59:53.293 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-09-10T18:59:53.293 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117209 s, 895 MB/s 2024-09-10T18:59:53.293 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-09-10T18:59:53.853 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:53 smithi194 ceph-mon[32464]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:53.853 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:53 smithi194 ceph-mon[32464]: from='osd.3 v2:172.21.15.119:6813/783850966' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-10T18:59:53.853 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:53 smithi194 ceph-mon[32464]: osdmap e24: 4 total, 3 up, 4 in 2024-09-10T18:59:53.854 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:53 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:53.854 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:53 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:53.880 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_4 2024-09-10T18:59:54.010 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:53 smithi119 ceph-mon[29577]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:54.010 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:53 smithi119 ceph-mon[29577]: from='osd.3 v2:172.21.15.119:6813/783850966' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-10T18:59:54.010 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:53 smithi119 ceph-mon[29577]: osdmap e24: 4 total, 3 up, 4 in 2024-09-10T18:59:54.010 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:53 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:54.010 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:53 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:54.010 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:53 smithi119 ceph-mon[40674]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:54.011 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:53 smithi119 ceph-mon[40674]: from='osd.3 v2:172.21.15.119:6813/783850966' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi119", "root=default"]}]': finished 2024-09-10T18:59:54.011 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:53 smithi119 ceph-mon[40674]: osdmap e24: 4 total, 3 up, 4 in 2024-09-10T18:59:54.011 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:53 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:54.011 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:53 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:54.122 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T18:59:54.942 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:54 smithi194 ceph-mon[32464]: purged_snaps scrub starts 2024-09-10T18:59:54.942 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:54 smithi194 ceph-mon[32464]: purged_snaps scrub ok 2024-09-10T18:59:54.942 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:54 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:54.942 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:54 smithi194 ceph-mon[32464]: osd.3 v2:172.21.15.119:6813/783850966 boot 2024-09-10T18:59:54.942 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:54 smithi194 ceph-mon[32464]: osdmap e25: 4 total, 4 up, 4 in 2024-09-10T18:59:54.943 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:54 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:54.973 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:54 smithi119 ceph-mon[29577]: purged_snaps scrub starts 2024-09-10T18:59:54.973 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:54 smithi119 ceph-mon[29577]: purged_snaps scrub ok 2024-09-10T18:59:54.973 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:54 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:54.973 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:54 smithi119 ceph-mon[29577]: osd.3 v2:172.21.15.119:6813/783850966 boot 2024-09-10T18:59:54.973 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:54 smithi119 ceph-mon[29577]: osdmap e25: 4 total, 4 up, 4 in 2024-09-10T18:59:54.973 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:54 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:54.973 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:54 smithi119 ceph-mon[40674]: purged_snaps scrub starts 2024-09-10T18:59:54.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:54 smithi119 ceph-mon[40674]: purged_snaps scrub ok 2024-09-10T18:59:54.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:54 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:54.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:54 smithi119 ceph-mon[40674]: osd.3 v2:172.21.15.119:6813/783850966 boot 2024-09-10T18:59:54.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:54 smithi119 ceph-mon[40674]: osdmap e25: 4 total, 4 up, 4 in 2024-09-10T18:59:54.974 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:54 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T18:59:55.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.641+0000 7f41e29b4640 1 -- 172.21.15.194:0/818949022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc0fe4d0 msgr2=0x7f41dc104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:55.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.641+0000 7f41e29b4640 1 --2- 172.21.15.194:0/818949022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc0fe4d0 0x7f41dc104d70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f41d4009cd0 tx=0x7f41d4004d10 comp rx=0 tx=0).stop 2024-09-10T18:59:55.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.641+0000 7f41e29b4640 1 -- 172.21.15.194:0/818949022 shutdown_connections 2024-09-10T18:59:55.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.641+0000 7f41e29b4640 1 --2- 172.21.15.194:0/818949022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc0fe4d0 0x7f41dc104d70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:55.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.641+0000 7f41e29b4640 1 --2- 172.21.15.194:0/818949022 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f41dc0fdb20 0x7f41dc0fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:55.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.641+0000 7f41e29b4640 1 --2- 172.21.15.194:0/818949022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41dc0fc850 0x7f41dc0fcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:55.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.641+0000 7f41e29b4640 1 -- 172.21.15.194:0/818949022 >> 172.21.15.194:0/818949022 conn(0x7f41dc0f8010 msgr2=0x7f41dc0fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T18:59:55.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.641+0000 7f41e29b4640 1 -- 172.21.15.194:0/818949022 shutdown_connections 2024-09-10T18:59:55.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.641+0000 7f41e29b4640 1 -- 172.21.15.194:0/818949022 wait complete. 2024-09-10T18:59:55.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41e29b4640 1 Processor -- start 2024-09-10T18:59:55.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41e29b4640 1 -- start start 2024-09-10T18:59:55.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41e29b4640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc0fc850 0x7f41dc10cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:59:55.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41e29b4640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f41dc0fdb20 0x7f41dc10d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:59:55.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41e29b4640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41dc0fe4d0 0x7f41dc114090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:59:55.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41e29b4640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc114660 con 0x7f41dc0fe4d0 2024-09-10T18:59:55.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41e29b4640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc1147d0 con 0x7f41dc0fc850 2024-09-10T18:59:55.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41e29b4640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41dc114990 con 0x7f41dc0fdb20 2024-09-10T18:59:55.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41dbfff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc0fc850 0x7f41dc10cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:59:55.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41dbfff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc0fc850 0x7f41dc10cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:44902/0 (socket says 172.21.15.194:44902) 2024-09-10T18:59:55.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41dbfff640 1 -- 172.21.15.194:0/157977782 learned_addr learned my addr 172.21.15.194:0/157977782 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T18:59:55.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41db7fe640 1 --2- 172.21.15.194:0/157977782 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f41dc0fdb20 0x7f41dc10d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:59:55.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41e0f51640 1 --2- 172.21.15.194:0/157977782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41dc0fe4d0 0x7f41dc114090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:59:55.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41dbfff640 1 -- 172.21.15.194:0/157977782 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f41dc0fdb20 msgr2=0x7f41dc10d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:55.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41dbfff640 1 --2- 172.21.15.194:0/157977782 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f41dc0fdb20 0x7f41dc10d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:55.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41dbfff640 1 -- 172.21.15.194:0/157977782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41dc0fe4d0 msgr2=0x7f41dc114090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T18:59:55.647 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41dbfff640 1 --2- 172.21.15.194:0/157977782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41dc0fe4d0 0x7f41dc114090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T18:59:55.647 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.642+0000 7f41dbfff640 1 -- 172.21.15.194:0/157977782 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d40098b0 con 0x7f41dc0fc850 2024-09-10T18:59:55.647 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.643+0000 7f41dbfff640 1 --2- 172.21.15.194:0/157977782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc0fc850 0x7f41dc10cb40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f41cc002ad0 tx=0x7f41cc004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:59:55.647 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.643+0000 7f41d8ff9640 1 -- 172.21.15.194:0/157977782 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc007c30 con 0x7f41dc0fc850 2024-09-10T18:59:55.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.643+0000 7f41d8ff9640 1 -- 172.21.15.194:0/157977782 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f41cc00f040 con 0x7f41dc0fc850 2024-09-10T18:59:55.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.643+0000 7f41e29b4640 1 -- 172.21.15.194:0/157977782 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41dc114c70 con 0x7f41dc0fc850 2024-09-10T18:59:55.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.643+0000 7f41d8ff9640 1 -- 172.21.15.194:0/157977782 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc00d8a0 con 0x7f41dc0fc850 2024-09-10T18:59:55.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.643+0000 7f41e29b4640 1 -- 172.21.15.194:0/157977782 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41dc1150f0 con 0x7f41dc0fc850 2024-09-10T18:59:55.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.644+0000 7f41e29b4640 1 -- 172.21.15.194:0/157977782 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c8005040 con 0x7f41dc0fc850 2024-09-10T18:59:55.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.645+0000 7f41d8ff9640 1 -- 172.21.15.194:0/157977782 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 15) v1 ==== 90746+0+0 (secure 0 0 0) 0x7f41cc00dc10 con 0x7f41dc0fc850 2024-09-10T18:59:55.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.645+0000 7f41d8ff9640 1 --2- 172.21.15.194:0/157977782 >> v2:172.21.15.119:6800/1094129222 conn(0x7f41c406ee10 0x7f41c40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T18:59:55.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.645+0000 7f41d8ff9640 1 -- 172.21.15.194:0/157977782 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 2799+0+0 (secure 0 0 0) 0x7f41cc08da00 con 0x7f41dc0fc850 2024-09-10T18:59:55.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.646+0000 7f41db7fe640 1 --2- 172.21.15.194:0/157977782 >> v2:172.21.15.119:6800/1094129222 conn(0x7f41c406ee10 0x7f41c40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T18:59:55.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.646+0000 7f41db7fe640 1 --2- 172.21.15.194:0/157977782 >> v2:172.21.15.119:6800/1094129222 conn(0x7f41c406ee10 0x7f41c40712c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f41dc10dff0 tx=0x7f41d000b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T18:59:55.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.649+0000 7f41d8ff9640 1 -- 172.21.15.194:0/157977782 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f41cc05b610 con 0x7f41dc0fc850 2024-09-10T18:59:55.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:55.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: osdmap e26: 4 total, 4 up, 4 in 2024-09-10T18:59:55.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:55.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:55.814 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T18:59:55.813+0000 7f41e29b4640 1 -- 172.21.15.194:0/157977782 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f41c8000bf0 con 0x7f41c406ee10 2024-09-10T18:59:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: osdmap e26: 4 total, 4 up, 4 in 2024-09-10T18:59:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:56.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:56.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:56.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:56.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:56.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T18:59:56.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: osdmap e26: 4 total, 4 up, 4 in 2024-09-10T18:59:56.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:56.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:56.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:56.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:56.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:56.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T18:59:56.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:56.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:56.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:56.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:56.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T18:59:56.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:56.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T18:59:56.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T18:59:56.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:56 smithi194 ceph-mon[32464]: Detected new or changed devices on smithi119 2024-09-10T18:59:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:56 smithi194 ceph-mon[32464]: Adjusting osd_memory_target on smithi119 to 4016M 2024-09-10T18:59:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:56 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:59:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:56 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:59:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:56 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:57.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:56 smithi119 ceph-mon[29577]: Detected new or changed devices on smithi119 2024-09-10T18:59:57.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:56 smithi119 ceph-mon[29577]: Adjusting osd_memory_target on smithi119 to 4016M 2024-09-10T18:59:57.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:56 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:59:57.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:56 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:59:57.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:56 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:57.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:56 smithi119 ceph-mon[40674]: Detected new or changed devices on smithi119 2024-09-10T18:59:57.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:56 smithi119 ceph-mon[40674]: Adjusting osd_memory_target on smithi119 to 4016M 2024-09-10T18:59:57.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:56 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T18:59:57.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:56 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T18:59:57.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:56 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T18:59:58.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:57 smithi194 ceph-mon[32464]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:59:58.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:57 smithi194 ceph-mon[32464]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:59:58.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:57 smithi119 ceph-mon[29577]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:59:58.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:57 smithi119 ceph-mon[29577]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:59:58.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:57 smithi119 ceph-mon[40674]: from='client.24205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T18:59:58.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:57 smithi119 ceph-mon[40674]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T18:59:59.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/1465463584' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bee9361-6ab4-4545-a6d4-b05bbb4f9c51"}]: dispatch 2024-09-10T18:59:59.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bee9361-6ab4-4545-a6d4-b05bbb4f9c51"}]: dispatch 2024-09-10T18:59:59.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bee9361-6ab4-4545-a6d4-b05bbb4f9c51"}]': finished 2024-09-10T18:59:59.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:58 smithi194 ceph-mon[32464]: osdmap e27: 5 total, 4 up, 5 in 2024-09-10T18:59:59.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:58 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:59:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/1008246835' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:59:59.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/1465463584' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bee9361-6ab4-4545-a6d4-b05bbb4f9c51"}]: dispatch 2024-09-10T18:59:59.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bee9361-6ab4-4545-a6d4-b05bbb4f9c51"}]: dispatch 2024-09-10T18:59:59.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bee9361-6ab4-4545-a6d4-b05bbb4f9c51"}]': finished 2024-09-10T18:59:59.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:58 smithi119 ceph-mon[29577]: osdmap e27: 5 total, 4 up, 5 in 2024-09-10T18:59:59.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:58 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:59:59.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/1008246835' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:59:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/1465463584' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bee9361-6ab4-4545-a6d4-b05bbb4f9c51"}]: dispatch 2024-09-10T18:59:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bee9361-6ab4-4545-a6d4-b05bbb4f9c51"}]: dispatch 2024-09-10T18:59:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bee9361-6ab4-4545-a6d4-b05bbb4f9c51"}]': finished 2024-09-10T18:59:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:58 smithi119 ceph-mon[40674]: osdmap e27: 5 total, 4 up, 5 in 2024-09-10T18:59:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:58 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T18:59:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/1008246835' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T18:59:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 18:59:59 smithi119 ceph-mon[29577]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:00.073 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 18:59:59 smithi119 ceph-mon[40674]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:00.073 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 18:59:59 smithi194 ceph-mon[32464]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:00.950 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:00 smithi194 ceph-mon[32464]: overall HEALTH_OK 2024-09-10T19:00:01.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:00 smithi119 ceph-mon[29577]: overall HEALTH_OK 2024-09-10T19:00:01.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:00 smithi119 ceph-mon[40674]: overall HEALTH_OK 2024-09-10T19:00:01.944 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:01 smithi194 ceph-mon[32464]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:02.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:01 smithi119 ceph-mon[29577]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:02.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:01 smithi119 ceph-mon[40674]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:03.736 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:03 smithi194 ceph-mon[32464]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:04.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:03 smithi119 ceph-mon[29577]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:04.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:03 smithi119 ceph-mon[40674]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:05.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-10T19:00:05.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:05.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-10T19:00:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-10T19:00:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:05.956 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:05 smithi194 ceph-mon[32464]: Deploying daemon osd.4 on smithi194 2024-09-10T19:00:05.956 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:05 smithi194 ceph-mon[32464]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:06.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:05 smithi119 ceph-mon[29577]: Deploying daemon osd.4 on smithi194 2024-09-10T19:00:06.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:05 smithi119 ceph-mon[29577]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:06.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:05 smithi119 ceph-mon[40674]: Deploying daemon osd.4 on smithi194 2024-09-10T19:00:06.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:05 smithi119 ceph-mon[40674]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:07.964 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:07 smithi194 ceph-mon[32464]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:08.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:07 smithi119 ceph-mon[29577]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:08.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:07 smithi119 ceph-mon[40674]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:08.975 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:08.975 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:08.975 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:08 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:09.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:09.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:09.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:08 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:09.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:09.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:09.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:08 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:10.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:09 smithi194 ceph-mon[32464]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:10.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:09 smithi119 ceph-mon[29577]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:10.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:09 smithi119 ceph-mon[40674]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:11.837 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:11 smithi194 ceph-mon[32464]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:11.837 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:11 smithi194 ceph-mon[32464]: from='osd.4 v2:172.21.15.194:6800/4259870591' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T19:00:11.837 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:11 smithi194 ceph-mon[32464]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T19:00:11.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:11.835+0000 7f41d8ff9640 1 -- 172.21.15.194:0/157977782 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f41c8000bf0 con 0x7f41c406ee10 2024-09-10T19:00:11.838 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 4 on host 'smithi194' 2024-09-10T19:00:11.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:11.838+0000 7f41e29b4640 1 -- 172.21.15.194:0/157977782 >> v2:172.21.15.119:6800/1094129222 conn(0x7f41c406ee10 msgr2=0x7f41c40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:11.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:11.838+0000 7f41e29b4640 1 --2- 172.21.15.194:0/157977782 >> v2:172.21.15.119:6800/1094129222 conn(0x7f41c406ee10 0x7f41c40712c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f41dc10dff0 tx=0x7f41d000b040 comp rx=0 tx=0).stop 2024-09-10T19:00:11.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:11.838+0000 7f41e29b4640 1 -- 172.21.15.194:0/157977782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc0fc850 msgr2=0x7f41dc10cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:11.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:11.838+0000 7f41e29b4640 1 --2- 172.21.15.194:0/157977782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc0fc850 0x7f41dc10cb40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f41cc002ad0 tx=0x7f41cc004600 comp rx=0 tx=0).stop 2024-09-10T19:00:11.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:11.838+0000 7f41e29b4640 1 -- 172.21.15.194:0/157977782 shutdown_connections 2024-09-10T19:00:11.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:11.838+0000 7f41e29b4640 1 --2- 172.21.15.194:0/157977782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41dc0fe4d0 0x7f41dc114090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:11.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:11.838+0000 7f41e29b4640 1 --2- 172.21.15.194:0/157977782 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f41dc0fdb20 0x7f41dc10d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:11.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:11.838+0000 7f41e29b4640 1 --2- 172.21.15.194:0/157977782 >> v2:172.21.15.119:6800/1094129222 conn(0x7f41c406ee10 0x7f41c40712c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:11.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:11.838+0000 7f41e29b4640 1 --2- 172.21.15.194:0/157977782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41dc0fc850 0x7f41dc10cb40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:11.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:11.838+0000 7f41e29b4640 1 -- 172.21.15.194:0/157977782 >> 172.21.15.194:0/157977782 conn(0x7f41dc0f8010 msgr2=0x7f41dc0f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:11.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:11.838+0000 7f41e29b4640 1 -- 172.21.15.194:0/157977782 shutdown_connections 2024-09-10T19:00:11.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:11.838+0000 7f41e29b4640 1 -- 172.21.15.194:0/157977782 wait complete. 2024-09-10T19:00:12.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:11 smithi119 ceph-mon[29577]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:12.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:11 smithi119 ceph-mon[29577]: from='osd.4 v2:172.21.15.194:6800/4259870591' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T19:00:12.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:11 smithi119 ceph-mon[29577]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T19:00:12.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:11 smithi119 ceph-mon[40674]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:12.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:11 smithi119 ceph-mon[40674]: from='osd.4 v2:172.21.15.194:6800/4259870591' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T19:00:12.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:11 smithi119 ceph-mon[40674]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T19:00:12.391 DEBUG:teuthology.orchestra.run.smithi194:osd.4> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.4.service 2024-09-10T19:00:12.393 INFO:tasks.cephadm:Deploying osd.5 on smithi194 with /dev/vg_nvme/lv_3... 2024-09-10T19:00:12.393 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-10T19:00:12.681 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:00:13.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:12 smithi194 ceph-mon[32464]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-10T19:00:13.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:12 smithi194 ceph-mon[32464]: osdmap e28: 5 total, 4 up, 5 in 2024-09-10T19:00:13.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:12 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:00:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:12 smithi194 ceph-mon[32464]: from='osd.4 v2:172.21.15.194:6800/4259870591' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:00:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:12 smithi194 ceph-mon[32464]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:00:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:12 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:12 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:12 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:12 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:12 smithi194 ceph-mon[32464]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:13.019 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:00:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:00:12.734+0000 7f0897695640 -1 osd.4 0 waiting for initial osdmap 2024-09-10T19:00:13.019 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:00:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:00:12.738+0000 7f0892ccd640 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:00:13.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[29577]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-10T19:00:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[29577]: osdmap e28: 5 total, 4 up, 5 in 2024-09-10T19:00:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:00:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[29577]: from='osd.4 v2:172.21.15.194:6800/4259870591' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:00:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[29577]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:00:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[29577]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:13.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[40674]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-10T19:00:13.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[40674]: osdmap e28: 5 total, 4 up, 5 in 2024-09-10T19:00:13.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:00:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[40674]: from='osd.4 v2:172.21.15.194:6800/4259870591' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:00:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[40674]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:00:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:12 smithi119 ceph-mon[40674]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:00:14.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:13 smithi194 ceph-mon[32464]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-10T19:00:14.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:13 smithi194 ceph-mon[32464]: osdmap e29: 5 total, 4 up, 5 in 2024-09-10T19:00:14.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:13 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:00:14.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:13 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:00:14.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:13 smithi119 ceph-mon[29577]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-10T19:00:14.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:13 smithi119 ceph-mon[29577]: osdmap e29: 5 total, 4 up, 5 in 2024-09-10T19:00:14.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:13 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:00:14.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:13 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:00:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:13 smithi119 ceph-mon[40674]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-10T19:00:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:13 smithi119 ceph-mon[40674]: osdmap e29: 5 total, 4 up, 5 in 2024-09-10T19:00:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:13 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:00:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:13 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:00:14.631 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-10T19:00:14.724 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-10T19:00:14.741 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-09-10T19:00:14.741 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-09-10T19:00:14.741 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0149875 s, 700 MB/s 2024-09-10T19:00:14.741 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-09-10T19:00:14.989 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:14 smithi194 ceph-mon[32464]: purged_snaps scrub starts 2024-09-10T19:00:14.989 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:14 smithi194 ceph-mon[32464]: purged_snaps scrub ok 2024-09-10T19:00:14.989 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:14 smithi194 ceph-mon[32464]: osd.4 v2:172.21.15.194:6800/4259870591 boot 2024-09-10T19:00:14.989 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:14 smithi194 ceph-mon[32464]: osdmap e30: 5 total, 5 up, 5 in 2024-09-10T19:00:14.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:14 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:00:14.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:14 smithi194 ceph-mon[32464]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:14.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:14 smithi194 ceph-mon[32464]: osdmap e31: 5 total, 5 up, 5 in 2024-09-10T19:00:15.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:14 smithi119 ceph-mon[29577]: purged_snaps scrub starts 2024-09-10T19:00:15.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:14 smithi119 ceph-mon[29577]: purged_snaps scrub ok 2024-09-10T19:00:15.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:14 smithi119 ceph-mon[29577]: osd.4 v2:172.21.15.194:6800/4259870591 boot 2024-09-10T19:00:15.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:14 smithi119 ceph-mon[29577]: osdmap e30: 5 total, 5 up, 5 in 2024-09-10T19:00:15.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:14 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:00:15.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:14 smithi119 ceph-mon[29577]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:15.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:14 smithi119 ceph-mon[29577]: osdmap e31: 5 total, 5 up, 5 in 2024-09-10T19:00:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:14 smithi119 ceph-mon[40674]: purged_snaps scrub starts 2024-09-10T19:00:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:14 smithi119 ceph-mon[40674]: purged_snaps scrub ok 2024-09-10T19:00:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:14 smithi119 ceph-mon[40674]: osd.4 v2:172.21.15.194:6800/4259870591 boot 2024-09-10T19:00:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:14 smithi119 ceph-mon[40674]: osdmap e30: 5 total, 5 up, 5 in 2024-09-10T19:00:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:14 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:00:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:14 smithi119 ceph-mon[40674]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:14 smithi119 ceph-mon[40674]: osdmap e31: 5 total, 5 up, 5 in 2024-09-10T19:00:15.549 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_3 2024-09-10T19:00:16.525 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:00:16.933 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:16 smithi194 ceph-mon[32464]: osdmap e32: 5 total, 5 up, 5 in 2024-09-10T19:00:16.933 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:16 smithi194 ceph-mon[32464]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:17.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:16 smithi119 ceph-mon[29577]: osdmap e32: 5 total, 5 up, 5 in 2024-09-10T19:00:17.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:16 smithi119 ceph-mon[29577]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:17.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:16 smithi119 ceph-mon[40674]: osdmap e32: 5 total, 5 up, 5 in 2024-09-10T19:00:17.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:16 smithi119 ceph-mon[40674]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:18.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.487+0000 7fc123367640 1 -- 172.21.15.194:0/799447766 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc11c0fe4e0 msgr2=0x7fc11c104d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:18.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.487+0000 7fc123367640 1 --2- 172.21.15.194:0/799447766 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc11c0fe4e0 0x7fc11c104d80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc114004950 tx=0x7fc114009da0 comp rx=0 tx=0).stop 2024-09-10T19:00:18.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 -- 172.21.15.194:0/799447766 shutdown_connections 2024-09-10T19:00:18.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 --2- 172.21.15.194:0/799447766 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc11c0fe4e0 0x7fc11c104d80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:18.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 --2- 172.21.15.194:0/799447766 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc11c0fdb30 0x7fc11c0fdfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:18.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 --2- 172.21.15.194:0/799447766 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc11c0fc860 0x7fc11c0fcc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:18.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 -- 172.21.15.194:0/799447766 >> 172.21.15.194:0/799447766 conn(0x7fc11c0f8040 msgr2=0x7fc11c0fa450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:18.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 -- 172.21.15.194:0/799447766 shutdown_connections 2024-09-10T19:00:18.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 -- 172.21.15.194:0/799447766 wait complete. 2024-09-10T19:00:18.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 Processor -- start 2024-09-10T19:00:18.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 -- start start 2024-09-10T19:00:18.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc11c0fc860 0x7fc11c10caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:00:18.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc11c0fdb30 0x7fc11c10cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:00:18.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc11c0fe4e0 0x7fc11c114040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:00:18.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c114610 con 0x7fc11c0fdb30 2024-09-10T19:00:18.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c114780 con 0x7fc11c0fe4e0 2024-09-10T19:00:18.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.488+0000 7fc123367640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc11c114940 con 0x7fc11c0fc860 2024-09-10T19:00:18.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc121904640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc11c0fe4e0 0x7fc11c114040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:00:18.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc121904640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc11c0fe4e0 0x7fc11c114040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:54542/0 (socket says 172.21.15.194:54542) 2024-09-10T19:00:18.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc121904640 1 -- 172.21.15.194:0/460384132 learned_addr learned my addr 172.21.15.194:0/460384132 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T19:00:18.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc120902640 1 --2- 172.21.15.194:0/460384132 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc11c0fdb30 0x7fc11c10cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:00:18.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc121103640 1 --2- 172.21.15.194:0/460384132 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc11c0fc860 0x7fc11c10caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:00:18.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc121904640 1 -- 172.21.15.194:0/460384132 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc11c0fc860 msgr2=0x7fc11c10caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:18.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc121904640 1 --2- 172.21.15.194:0/460384132 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc11c0fc860 0x7fc11c10caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:18.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc121904640 1 -- 172.21.15.194:0/460384132 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc11c0fdb30 msgr2=0x7fc11c10cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:18.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc121904640 1 --2- 172.21.15.194:0/460384132 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc11c0fdb30 0x7fc11c10cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:18.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc121904640 1 -- 172.21.15.194:0/460384132 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc114005040 con 0x7fc11c0fe4e0 2024-09-10T19:00:18.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc121904640 1 --2- 172.21.15.194:0/460384132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc11c0fe4e0 0x7fc11c114040 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc11402d040 tx=0x7fc114035040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:00:18.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc119ffb640 1 -- 172.21.15.194:0/460384132 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11402ad70 con 0x7fc11c0fe4e0 2024-09-10T19:00:18.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc119ffb640 1 -- 172.21.15.194:0/460384132 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fc114032a50 con 0x7fc11c0fe4e0 2024-09-10T19:00:18.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc123367640 1 -- 172.21.15.194:0/460384132 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc11c114bc0 con 0x7fc11c0fe4e0 2024-09-10T19:00:18.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.489+0000 7fc119ffb640 1 -- 172.21.15.194:0/460384132 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1140337e0 con 0x7fc11c0fe4e0 2024-09-10T19:00:18.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.490+0000 7fc123367640 1 -- 172.21.15.194:0/460384132 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11c115090 con 0x7fc11c0fe4e0 2024-09-10T19:00:18.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.491+0000 7fc123367640 1 -- 172.21.15.194:0/460384132 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc11c04bf60 con 0x7fc11c0fe4e0 2024-09-10T19:00:18.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.491+0000 7fc119ffb640 1 -- 172.21.15.194:0/460384132 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 15) v1 ==== 90746+0+0 (secure 0 0 0) 0x7fc114030020 con 0x7fc11c0fe4e0 2024-09-10T19:00:18.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.491+0000 7fc119ffb640 1 --2- 172.21.15.194:0/460384132 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc10406ee10 0x7fc1040712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:00:18.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.492+0000 7fc119ffb640 1 -- 172.21.15.194:0/460384132 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 3201+0+0 (secure 0 0 0) 0x7fc1140ade60 con 0x7fc11c0fe4e0 2024-09-10T19:00:18.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.492+0000 7fc121103640 1 --2- 172.21.15.194:0/460384132 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc10406ee10 0x7fc1040712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:00:18.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.492+0000 7fc121103640 1 --2- 172.21.15.194:0/460384132 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc10406ee10 0x7fc1040712c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc10c004fd0 tx=0x7fc10c009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:00:18.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.496+0000 7fc119ffb640 1 -- 172.21.15.194:0/460384132 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fc11407b980 con 0x7fc11c0fe4e0 2024-09-10T19:00:18.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:18.656+0000 7fc123367640 1 -- 172.21.15.194:0/460384132 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc11c107800 con 0x7fc10406ee10 2024-09-10T19:00:19.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: Detected new or changed devices on smithi194 2024-09-10T19:00:19.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.369 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:19.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: Adjusting osd_memory_target on smithi194 to 17088M 2024-09-10T19:00:19.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:19.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:19.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-09-10T19:00:19.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:19.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:19.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:19.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:00:19.370 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:00:19.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:00:19.371 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:19.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: Detected new or changed devices on smithi194 2024-09-10T19:00:19.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:19.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: Adjusting osd_memory_target on smithi194 to 17088M 2024-09-10T19:00:19.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:19.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:19.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.371 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-09-10T19:00:19.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:19.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:19.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:19.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:00:19.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:00:19.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:00:19.372 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:19 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:19.403 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: Detected new or changed devices on smithi194 2024-09-10T19:00:19.403 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.403 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.403 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:19.403 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: Adjusting osd_memory_target on smithi194 to 17088M 2024-09-10T19:00:19.403 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.403 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:19.403 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:19.403 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.403 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-09-10T19:00:19.403 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:19.403 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:19.403 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:19.403 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:19.404 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:00:19.404 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:00:19.404 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:00:19.404 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:19 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:21.513 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:21 smithi194 ceph-mon[32464]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-09-10T19:00:21.513 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/2574522759' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b26af90f-5d90-4929-94e6-92545dffd78d"}]: dispatch 2024-09-10T19:00:21.514 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:21 smithi194 ceph-mon[32464]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b26af90f-5d90-4929-94e6-92545dffd78d"}]: dispatch 2024-09-10T19:00:21.514 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:21 smithi194 ceph-mon[32464]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b26af90f-5d90-4929-94e6-92545dffd78d"}]': finished 2024-09-10T19:00:21.514 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:21 smithi194 ceph-mon[32464]: osdmap e33: 6 total, 5 up, 6 in 2024-09-10T19:00:21.514 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:21 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:21.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:21 smithi119 ceph-mon[29577]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-09-10T19:00:21.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/2574522759' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b26af90f-5d90-4929-94e6-92545dffd78d"}]: dispatch 2024-09-10T19:00:21.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:21 smithi119 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b26af90f-5d90-4929-94e6-92545dffd78d"}]: dispatch 2024-09-10T19:00:21.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:21 smithi119 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b26af90f-5d90-4929-94e6-92545dffd78d"}]': finished 2024-09-10T19:00:21.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:21 smithi119 ceph-mon[29577]: osdmap e33: 6 total, 5 up, 6 in 2024-09-10T19:00:21.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:21 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:21.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:21 smithi119 ceph-mon[40674]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-09-10T19:00:21.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/2574522759' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b26af90f-5d90-4929-94e6-92545dffd78d"}]: dispatch 2024-09-10T19:00:21.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:21 smithi119 ceph-mon[40674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b26af90f-5d90-4929-94e6-92545dffd78d"}]: dispatch 2024-09-10T19:00:21.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:21 smithi119 ceph-mon[40674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b26af90f-5d90-4929-94e6-92545dffd78d"}]': finished 2024-09-10T19:00:21.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:21 smithi119 ceph-mon[40674]: osdmap e33: 6 total, 5 up, 6 in 2024-09-10T19:00:21.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:21 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:22.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/2554642942' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:00:22.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/2554642942' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:00:22.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/2554642942' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:00:23.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:23 smithi194 ceph-mon[32464]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-10T19:00:23.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:23 smithi119 ceph-mon[29577]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-10T19:00:23.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:23 smithi119 ceph-mon[40674]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-09-10T19:00:25.319 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:25 smithi194 ceph-mon[32464]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-10T19:00:25.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:25 smithi119 ceph-mon[29577]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-10T19:00:25.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:25 smithi119 ceph-mon[40674]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-10T19:00:27.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:27 smithi194 ceph-mon[32464]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:00:27.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:27 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-10T19:00:27.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:27 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:27.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:27 smithi119 ceph-mon[29577]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:00:27.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:27 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-10T19:00:27.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:27 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:27.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:27 smithi119 ceph-mon[40674]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:00:27.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:27 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-10T19:00:27.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:27 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:28.415 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:28 smithi194 ceph-mon[32464]: Deploying daemon osd.5 on smithi194 2024-09-10T19:00:28.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:28 smithi119 ceph-mon[29577]: Deploying daemon osd.5 on smithi194 2024-09-10T19:00:28.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:28 smithi119 ceph-mon[40674]: Deploying daemon osd.5 on smithi194 2024-09-10T19:00:29.305 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:29 smithi194 ceph-mon[32464]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:29.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:29 smithi119 ceph-mon[29577]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:29.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:29 smithi119 ceph-mon[40674]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:31.370 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:31 smithi194 ceph-mon[32464]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:31.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:31 smithi119 ceph-mon[29577]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:31.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:31 smithi119 ceph-mon[40674]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:32.290 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:32 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:32.290 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:32 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:32.290 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:32 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:32.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:32 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:32.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:32 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:32.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:32 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:32.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:32 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:32.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:32 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:32.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:32 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:33.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:33 smithi194 ceph-mon[32464]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:34.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:33 smithi119 ceph-mon[29577]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:34.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:33 smithi119 ceph-mon[40674]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:34.662 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:34.661+0000 7fc119ffb640 1 -- 172.21.15.194:0/460384132 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc11c107800 con 0x7fc10406ee10 2024-09-10T19:00:34.663 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 5 on host 'smithi194' 2024-09-10T19:00:34.665 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:34.664+0000 7fc123367640 1 -- 172.21.15.194:0/460384132 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc10406ee10 msgr2=0x7fc1040712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:34.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:34.664+0000 7fc123367640 1 --2- 172.21.15.194:0/460384132 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc10406ee10 0x7fc1040712c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc10c004fd0 tx=0x7fc10c009040 comp rx=0 tx=0).stop 2024-09-10T19:00:34.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:34.664+0000 7fc123367640 1 -- 172.21.15.194:0/460384132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc11c0fe4e0 msgr2=0x7fc11c114040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:34.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:34.664+0000 7fc123367640 1 --2- 172.21.15.194:0/460384132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc11c0fe4e0 0x7fc11c114040 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc11402d040 tx=0x7fc114035040 comp rx=0 tx=0).stop 2024-09-10T19:00:34.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:34.664+0000 7fc123367640 1 -- 172.21.15.194:0/460384132 shutdown_connections 2024-09-10T19:00:34.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:34.664+0000 7fc123367640 1 --2- 172.21.15.194:0/460384132 >> v2:172.21.15.119:6800/1094129222 conn(0x7fc10406ee10 0x7fc1040712c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:34.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:34.664+0000 7fc123367640 1 --2- 172.21.15.194:0/460384132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc11c0fe4e0 0x7fc11c114040 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:34.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:34.664+0000 7fc123367640 1 --2- 172.21.15.194:0/460384132 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc11c0fdb30 0x7fc11c10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:34.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:34.664+0000 7fc123367640 1 --2- 172.21.15.194:0/460384132 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc11c0fc860 0x7fc11c10caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:34.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:34.664+0000 7fc123367640 1 -- 172.21.15.194:0/460384132 >> 172.21.15.194:0/460384132 conn(0x7fc11c0f8040 msgr2=0x7fc11c0f9b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:34.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:34.664+0000 7fc123367640 1 -- 172.21.15.194:0/460384132 shutdown_connections 2024-09-10T19:00:34.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:34.664+0000 7fc123367640 1 -- 172.21.15.194:0/460384132 wait complete. 2024-09-10T19:00:35.306 DEBUG:teuthology.orchestra.run.smithi194:osd.5> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.5.service 2024-09-10T19:00:35.308 INFO:tasks.cephadm:Deploying osd.6 on smithi194 with /dev/vg_nvme/lv_2... 2024-09-10T19:00:35.308 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-10T19:00:35.594 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:00:35.983 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:35 smithi194 ceph-mon[32464]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:35.984 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:35 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:35.984 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:35 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:35.984 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:35 smithi194 ceph-mon[32464]: from='osd.5 v2:172.21.15.194:6804/635822671' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T19:00:35.984 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:35 smithi194 ceph-mon[32464]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T19:00:35.984 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:35 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:35.984 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:35 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:36.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:35 smithi119 ceph-mon[29577]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:36.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:35 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:36.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:35 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:36.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:35 smithi119 ceph-mon[29577]: from='osd.5 v2:172.21.15.194:6804/635822671' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T19:00:36.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:35 smithi119 ceph-mon[29577]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T19:00:36.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:35 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:36.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:35 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:36.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:35 smithi119 ceph-mon[40674]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:36.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:35 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:36.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:35 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:36.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:35 smithi119 ceph-mon[40674]: from='osd.5 v2:172.21.15.194:6804/635822671' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T19:00:36.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:35 smithi119 ceph-mon[40674]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T19:00:36.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:35 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:36.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:35 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:36.965 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:36 smithi194 ceph-mon[32464]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-10T19:00:36.965 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:36 smithi194 ceph-mon[32464]: osdmap e34: 6 total, 5 up, 6 in 2024-09-10T19:00:36.965 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:36 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:36.965 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:36 smithi194 ceph-mon[32464]: from='osd.5 v2:172.21.15.194:6804/635822671' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:00:36.966 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:36 smithi194 ceph-mon[32464]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:00:36.966 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:00:36 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:00:36.676+0000 7f2e257be640 -1 osd.5 0 waiting for initial osdmap 2024-09-10T19:00:36.966 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:00:36 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:00:36.681+0000 7f2e215f7640 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:00:37.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:36 smithi119 ceph-mon[29577]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-10T19:00:37.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:36 smithi119 ceph-mon[29577]: osdmap e34: 6 total, 5 up, 6 in 2024-09-10T19:00:37.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:36 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:37.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:36 smithi119 ceph-mon[29577]: from='osd.5 v2:172.21.15.194:6804/635822671' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:00:37.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:36 smithi119 ceph-mon[29577]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:00:37.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:36 smithi119 ceph-mon[40674]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-10T19:00:37.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:36 smithi119 ceph-mon[40674]: osdmap e34: 6 total, 5 up, 6 in 2024-09-10T19:00:37.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:36 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:37.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:36 smithi119 ceph-mon[40674]: from='osd.5 v2:172.21.15.194:6804/635822671' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:00:37.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:36 smithi119 ceph-mon[40674]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:00:37.961 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:37 smithi194 ceph-mon[32464]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:37.961 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:37 smithi194 ceph-mon[32464]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-10T19:00:37.961 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:37 smithi194 ceph-mon[32464]: osdmap e35: 6 total, 5 up, 6 in 2024-09-10T19:00:37.962 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:37 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:37.962 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:37 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:38.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:37 smithi119 ceph-mon[29577]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:37 smithi119 ceph-mon[29577]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-10T19:00:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:37 smithi119 ceph-mon[29577]: osdmap e35: 6 total, 5 up, 6 in 2024-09-10T19:00:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:37 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:37 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:37 smithi119 ceph-mon[40674]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:00:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:37 smithi119 ceph-mon[40674]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-10T19:00:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:37 smithi119 ceph-mon[40674]: osdmap e35: 6 total, 5 up, 6 in 2024-09-10T19:00:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:37 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:37 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:38.131 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-10T19:00:38.228 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-10T19:00:38.246 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-09-10T19:00:38.246 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-09-10T19:00:38.246 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0155413 s, 675 MB/s 2024-09-10T19:00:38.246 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-09-10T19:00:38.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:38 smithi194 ceph-mon[32464]: purged_snaps scrub starts 2024-09-10T19:00:38.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:38 smithi194 ceph-mon[32464]: purged_snaps scrub ok 2024-09-10T19:00:38.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:38 smithi194 ceph-mon[32464]: osd.5 v2:172.21.15.194:6804/635822671 boot 2024-09-10T19:00:38.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:38 smithi194 ceph-mon[32464]: osdmap e36: 6 total, 6 up, 6 in 2024-09-10T19:00:38.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:38 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:38.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:38 smithi194 ceph-mon[32464]: osdmap e37: 6 total, 6 up, 6 in 2024-09-10T19:00:39.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:38 smithi119 ceph-mon[29577]: purged_snaps scrub starts 2024-09-10T19:00:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:38 smithi119 ceph-mon[29577]: purged_snaps scrub ok 2024-09-10T19:00:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:38 smithi119 ceph-mon[29577]: osd.5 v2:172.21.15.194:6804/635822671 boot 2024-09-10T19:00:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:38 smithi119 ceph-mon[29577]: osdmap e36: 6 total, 6 up, 6 in 2024-09-10T19:00:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:38 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:38 smithi119 ceph-mon[29577]: osdmap e37: 6 total, 6 up, 6 in 2024-09-10T19:00:39.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:38 smithi119 ceph-mon[40674]: purged_snaps scrub starts 2024-09-10T19:00:39.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:38 smithi119 ceph-mon[40674]: purged_snaps scrub ok 2024-09-10T19:00:39.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:38 smithi119 ceph-mon[40674]: osd.5 v2:172.21.15.194:6804/635822671 boot 2024-09-10T19:00:39.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:38 smithi119 ceph-mon[40674]: osdmap e36: 6 total, 6 up, 6 in 2024-09-10T19:00:39.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:38 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:00:39.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:38 smithi119 ceph-mon[40674]: osdmap e37: 6 total, 6 up, 6 in 2024-09-10T19:00:39.103 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_2 2024-09-10T19:00:39.795 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:00:39.981 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:39 smithi194 ceph-mon[32464]: pgmap v126: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:39.981 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:39 smithi194 ceph-mon[32464]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-10T19:00:40.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:39 smithi119 ceph-mon[29577]: pgmap v126: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:40.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:39 smithi119 ceph-mon[29577]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-10T19:00:40.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:39 smithi119 ceph-mon[40674]: pgmap v126: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:40.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:39 smithi119 ceph-mon[40674]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-10T19:00:40.911 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: osdmap e38: 6 total, 6 up, 6 in 2024-09-10T19:00:40.911 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: Detected new or changed devices on smithi194 2024-09-10T19:00:40.911 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:40.911 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:40.911 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:40.911 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:40.911 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: Adjusting osd_memory_target on smithi194 to 8544M 2024-09-10T19:00:40.911 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:40.911 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:40.911 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:40.912 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:40.912 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:40.912 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:40.912 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:40.912 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:40 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: osdmap e38: 6 total, 6 up, 6 in 2024-09-10T19:00:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: Detected new or changed devices on smithi194 2024-09-10T19:00:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: Adjusting osd_memory_target on smithi194 to 8544M 2024-09-10T19:00:41.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:41.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:41.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:41.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:41.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:41.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:41.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:41.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:41.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: osdmap e38: 6 total, 6 up, 6 in 2024-09-10T19:00:41.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: Detected new or changed devices on smithi194 2024-09-10T19:00:41.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:41.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:41.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:41.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:00:41.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: Adjusting osd_memory_target on smithi194 to 8544M 2024-09-10T19:00:41.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:41.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:41.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:41.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:41.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:41.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:41.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:00:41.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:40 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:41.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.350+0000 7efca3715640 1 -- 172.21.15.194:0/2497069375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc9c0fb740 msgr2=0x7efc9c0fbb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:41.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.350+0000 7efca3715640 1 --2- 172.21.15.194:0/2497069375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc9c0fb740 0x7efc9c0fbb50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efc98004950 tx=0x7efc98009da0 comp rx=0 tx=0).stop 2024-09-10T19:00:41.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 -- 172.21.15.194:0/2497069375 shutdown_connections 2024-09-10T19:00:41.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 --2- 172.21.15.194:0/2497069375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc9c0fd3c0 0x7efc9c103c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:41.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 --2- 172.21.15.194:0/2497069375 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efc9c0fca10 0x7efc9c0fce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:41.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 --2- 172.21.15.194:0/2497069375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc9c0fb740 0x7efc9c0fbb50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:41.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 -- 172.21.15.194:0/2497069375 >> 172.21.15.194:0/2497069375 conn(0x7efc9c0f6ee0 msgr2=0x7efc9c0f9330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:41.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 -- 172.21.15.194:0/2497069375 shutdown_connections 2024-09-10T19:00:41.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 -- 172.21.15.194:0/2497069375 wait complete. 2024-09-10T19:00:41.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 Processor -- start 2024-09-10T19:00:41.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 -- start start 2024-09-10T19:00:41.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efc9c0fb740 0x7efc9c10cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:00:41.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc9c0fca10 0x7efc9c10d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:00:41.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc9c0fd3c0 0x7efc9c114490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:00:41.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c114a60 con 0x7efc9c0fca10 2024-09-10T19:00:41.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c114bd0 con 0x7efc9c0fd3c0 2024-09-10T19:00:41.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca3715640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc9c114d90 con 0x7efc9c0fb740 2024-09-10T19:00:41.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca1cb2640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc9c0fd3c0 0x7efc9c114490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:00:41.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca1cb2640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc9c0fd3c0 0x7efc9c114490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:51580/0 (socket says 172.21.15.194:51580) 2024-09-10T19:00:41.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca1cb2640 1 -- 172.21.15.194:0/3329759498 learned_addr learned my addr 172.21.15.194:0/3329759498 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T19:00:41.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca0cb0640 1 --2- 172.21.15.194:0/3329759498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc9c0fca10 0x7efc9c10d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:00:41.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.351+0000 7efca14b1640 1 --2- 172.21.15.194:0/3329759498 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efc9c0fb740 0x7efc9c10cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:00:41.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.352+0000 7efca1cb2640 1 -- 172.21.15.194:0/3329759498 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efc9c0fb740 msgr2=0x7efc9c10cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:41.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.352+0000 7efca1cb2640 1 --2- 172.21.15.194:0/3329759498 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efc9c0fb740 0x7efc9c10cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:41.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.352+0000 7efca1cb2640 1 -- 172.21.15.194:0/3329759498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc9c0fca10 msgr2=0x7efc9c10d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:41.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.352+0000 7efca1cb2640 1 --2- 172.21.15.194:0/3329759498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc9c0fca10 0x7efc9c10d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:41.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.352+0000 7efca1cb2640 1 -- 172.21.15.194:0/3329759498 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc98005040 con 0x7efc9c0fd3c0 2024-09-10T19:00:41.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.352+0000 7efca1cb2640 1 --2- 172.21.15.194:0/3329759498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc9c0fd3c0 0x7efc9c114490 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efc9400bc30 tx=0x7efc94009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:00:41.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.352+0000 7efc91ffb640 1 -- 172.21.15.194:0/3329759498 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc940092e0 con 0x7efc9c0fd3c0 2024-09-10T19:00:41.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.352+0000 7efca3715640 1 -- 172.21.15.194:0/3329759498 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc9c115070 con 0x7efc9c0fd3c0 2024-09-10T19:00:41.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.352+0000 7efc91ffb640 1 -- 172.21.15.194:0/3329759498 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7efc9400ccd0 con 0x7efc9c0fd3c0 2024-09-10T19:00:41.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.352+0000 7efc91ffb640 1 -- 172.21.15.194:0/3329759498 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc94015d70 con 0x7efc9c0fd3c0 2024-09-10T19:00:41.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.352+0000 7efca3715640 1 -- 172.21.15.194:0/3329759498 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc9c1154f0 con 0x7efc9c0fd3c0 2024-09-10T19:00:41.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.353+0000 7efca3715640 1 -- 172.21.15.194:0/3329759498 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc9c04bf60 con 0x7efc9c0fd3c0 2024-09-10T19:00:41.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.354+0000 7efc91ffb640 1 -- 172.21.15.194:0/3329759498 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 15) v1 ==== 90746+0+0 (secure 0 0 0) 0x7efc94012070 con 0x7efc9c0fd3c0 2024-09-10T19:00:41.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.354+0000 7efc91ffb640 1 --2- 172.21.15.194:0/3329759498 >> v2:172.21.15.119:6800/1094129222 conn(0x7efc8806ee10 0x7efc880712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:00:41.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.354+0000 7efc91ffb640 1 -- 172.21.15.194:0/3329759498 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 3493+0+0 (secure 0 0 0) 0x7efc9408d660 con 0x7efc9c0fd3c0 2024-09-10T19:00:41.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.354+0000 7efca14b1640 1 --2- 172.21.15.194:0/3329759498 >> v2:172.21.15.119:6800/1094129222 conn(0x7efc8806ee10 0x7efc880712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:00:41.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.355+0000 7efca14b1640 1 --2- 172.21.15.194:0/3329759498 >> v2:172.21.15.119:6800/1094129222 conn(0x7efc8806ee10 0x7efc880712c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efc980094b0 tx=0x7efc98035040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:00:41.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.358+0000 7efc91ffb640 1 -- 172.21.15.194:0/3329759498 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7efc9405afe0 con 0x7efc9c0fd3c0 2024-09-10T19:00:41.521 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:41.520+0000 7efca3715640 1 -- 172.21.15.194:0/3329759498 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7efc9c106750 con 0x7efc8806ee10 2024-09-10T19:00:41.922 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:41 smithi194 ceph-mon[32464]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:41.922 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:41 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:00:41.922 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:41 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:00:41.923 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:41 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:42.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:41 smithi119 ceph-mon[29577]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:42.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:41 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:00:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:41 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:00:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:41 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:41 smithi119 ceph-mon[40674]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:41 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:00:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:41 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:00:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:41 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:42.944 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:42 smithi194 ceph-mon[32464]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:00:43.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:42 smithi119 ceph-mon[29577]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:00:43.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:42 smithi119 ceph-mon[40674]: from='client.24259 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:00:44.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:43 smithi194 ceph-mon[32464]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:44.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:43 smithi119 ceph-mon[29577]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:44.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:43 smithi119 ceph-mon[40674]: pgmap v130: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:45.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/2811905760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c05bb66f-8bc0-44b3-965d-fda246401108"}]: dispatch 2024-09-10T19:00:45.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c05bb66f-8bc0-44b3-965d-fda246401108"}]: dispatch 2024-09-10T19:00:45.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c05bb66f-8bc0-44b3-965d-fda246401108"}]': finished 2024-09-10T19:00:45.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:44 smithi194 ceph-mon[32464]: osdmap e39: 7 total, 6 up, 7 in 2024-09-10T19:00:45.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:44 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:00:45.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/2491800014' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:00:45.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/2811905760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c05bb66f-8bc0-44b3-965d-fda246401108"}]: dispatch 2024-09-10T19:00:45.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c05bb66f-8bc0-44b3-965d-fda246401108"}]: dispatch 2024-09-10T19:00:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c05bb66f-8bc0-44b3-965d-fda246401108"}]': finished 2024-09-10T19:00:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:44 smithi119 ceph-mon[29577]: osdmap e39: 7 total, 6 up, 7 in 2024-09-10T19:00:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:44 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:00:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/2491800014' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:00:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/2811905760' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c05bb66f-8bc0-44b3-965d-fda246401108"}]: dispatch 2024-09-10T19:00:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c05bb66f-8bc0-44b3-965d-fda246401108"}]: dispatch 2024-09-10T19:00:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c05bb66f-8bc0-44b3-965d-fda246401108"}]': finished 2024-09-10T19:00:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:44 smithi119 ceph-mon[40674]: osdmap e39: 7 total, 6 up, 7 in 2024-09-10T19:00:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:44 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:00:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/2491800014' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:00:46.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:45 smithi194 ceph-mon[32464]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T19:00:46.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:45 smithi194 ceph-mon[32464]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-10T19:00:46.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:45 smithi194 ceph-mon[32464]: Cluster is now healthy 2024-09-10T19:00:46.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:45 smithi119 ceph-mon[29577]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T19:00:46.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:45 smithi119 ceph-mon[29577]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-10T19:00:46.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:45 smithi119 ceph-mon[29577]: Cluster is now healthy 2024-09-10T19:00:46.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:45 smithi119 ceph-mon[40674]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T19:00:46.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:45 smithi119 ceph-mon[40674]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-10T19:00:46.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:45 smithi119 ceph-mon[40674]: Cluster is now healthy 2024-09-10T19:00:47.962 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:47 smithi194 ceph-mon[32464]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-10T19:00:48.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:47 smithi119 ceph-mon[29577]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-10T19:00:48.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:47 smithi119 ceph-mon[40674]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-09-10T19:00:49.938 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:49 smithi194 ceph-mon[32464]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-10T19:00:50.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:49 smithi119 ceph-mon[29577]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-10T19:00:50.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:49 smithi119 ceph-mon[40674]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-10T19:00:51.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:50 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-10T19:00:51.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:50 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:51.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:50 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-10T19:00:51.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:50 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:51.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:50 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-10T19:00:51.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:50 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:00:51.870 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:51 smithi194 ceph-mon[32464]: Deploying daemon osd.6 on smithi194 2024-09-10T19:00:51.870 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:51 smithi194 ceph-mon[32464]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:00:52.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:51 smithi119 ceph-mon[29577]: Deploying daemon osd.6 on smithi194 2024-09-10T19:00:52.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:51 smithi119 ceph-mon[29577]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:00:52.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:51 smithi119 ceph-mon[40674]: Deploying daemon osd.6 on smithi194 2024-09-10T19:00:52.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:51 smithi119 ceph-mon[40674]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:00:54.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:53 smithi194 ceph-mon[32464]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:00:54.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:53 smithi119 ceph-mon[29577]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:00:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:53 smithi119 ceph-mon[40674]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:00:55.714 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:55 smithi119 ceph-mon[29577]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:55.714 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:55.714 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:55.714 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:55.714 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:55 smithi119 ceph-mon[40674]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:55.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:55 smithi194 ceph-mon[32464]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:55.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:55.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:55.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:56.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:00:56.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:56.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:57.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:57 smithi194 ceph-mon[32464]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:58.012 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:58.010+0000 7efc91ffb640 1 -- 172.21.15.194:0/3329759498 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efc9c106750 con 0x7efc8806ee10 2024-09-10T19:00:58.012 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 6 on host 'smithi194' 2024-09-10T19:00:58.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:58.013+0000 7efca3715640 1 -- 172.21.15.194:0/3329759498 >> v2:172.21.15.119:6800/1094129222 conn(0x7efc8806ee10 msgr2=0x7efc880712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:58.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:58.013+0000 7efca3715640 1 --2- 172.21.15.194:0/3329759498 >> v2:172.21.15.119:6800/1094129222 conn(0x7efc8806ee10 0x7efc880712c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7efc980094b0 tx=0x7efc98035040 comp rx=0 tx=0).stop 2024-09-10T19:00:58.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:58.013+0000 7efca3715640 1 -- 172.21.15.194:0/3329759498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc9c0fd3c0 msgr2=0x7efc9c114490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:00:58.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:58.013+0000 7efca3715640 1 --2- 172.21.15.194:0/3329759498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc9c0fd3c0 0x7efc9c114490 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efc9400bc30 tx=0x7efc94009840 comp rx=0 tx=0).stop 2024-09-10T19:00:58.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:58.013+0000 7efca3715640 1 -- 172.21.15.194:0/3329759498 shutdown_connections 2024-09-10T19:00:58.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:58.013+0000 7efca3715640 1 --2- 172.21.15.194:0/3329759498 >> v2:172.21.15.119:6800/1094129222 conn(0x7efc8806ee10 0x7efc880712c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:58.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:58.013+0000 7efca3715640 1 --2- 172.21.15.194:0/3329759498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc9c0fd3c0 0x7efc9c114490 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:58.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:58.013+0000 7efca3715640 1 --2- 172.21.15.194:0/3329759498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc9c0fca10 0x7efc9c10d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:58.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:58.013+0000 7efca3715640 1 --2- 172.21.15.194:0/3329759498 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efc9c0fb740 0x7efc9c10cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:00:58.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:58.013+0000 7efca3715640 1 -- 172.21.15.194:0/3329759498 >> 172.21.15.194:0/3329759498 conn(0x7efc9c0f6ee0 msgr2=0x7efc9c066860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:00:58.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:58.013+0000 7efca3715640 1 -- 172.21.15.194:0/3329759498 shutdown_connections 2024-09-10T19:00:58.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:00:58.013+0000 7efca3715640 1 -- 172.21.15.194:0/3329759498 wait complete. 2024-09-10T19:00:58.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:57 smithi119 ceph-mon[40674]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:58.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:57 smithi119 ceph-mon[29577]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:58.578 DEBUG:teuthology.orchestra.run.smithi194:osd.6> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.6.service 2024-09-10T19:00:58.581 INFO:tasks.cephadm:Deploying osd.7 on smithi194 with /dev/vg_nvme/lv_1... 2024-09-10T19:00:58.581 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-10T19:00:58.881 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:00:59.003 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:58 smithi194 ceph-mon[32464]: from='osd.6 v2:172.21.15.194:6808/2830881352' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T19:00:59.003 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:58 smithi194 ceph-mon[32464]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T19:00:59.003 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:58 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:59.003 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:58 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:59.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:58 smithi119 ceph-mon[29577]: from='osd.6 v2:172.21.15.194:6808/2830881352' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T19:00:59.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:58 smithi119 ceph-mon[29577]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T19:00:59.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:58 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:59.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:58 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:58 smithi119 ceph-mon[40674]: from='osd.6 v2:172.21.15.194:6808/2830881352' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T19:00:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:58 smithi119 ceph-mon[40674]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T19:00:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:58 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:58 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:59.929 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:59 smithi194 ceph-mon[32464]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:00:59.929 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:59 smithi194 ceph-mon[32464]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T19:00:59.929 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:59 smithi194 ceph-mon[32464]: osdmap e40: 7 total, 6 up, 7 in 2024-09-10T19:00:59.929 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:59 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:00:59.929 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:59 smithi194 ceph-mon[32464]: from='osd.6 v2:172.21.15.194:6808/2830881352' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:00:59.929 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:59 smithi194 ceph-mon[32464]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:00:59.929 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:59 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:59.929 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:00:59 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:00:59.930 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:00:59 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:00:59.741+0000 7f3c324e0640 -1 osd.6 0 waiting for initial osdmap 2024-09-10T19:00:59.930 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:00:59 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:00:59.746+0000 7f3c2f31b640 -1 osd.6 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:01:00.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[29577]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:01:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[29577]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T19:01:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[29577]: osdmap e40: 7 total, 6 up, 7 in 2024-09-10T19:01:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:01:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[29577]: from='osd.6 v2:172.21.15.194:6808/2830881352' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:01:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[29577]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:01:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[40674]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:01:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[40674]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T19:01:00.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[40674]: osdmap e40: 7 total, 6 up, 7 in 2024-09-10T19:01:00.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:01:00.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[40674]: from='osd.6 v2:172.21.15.194:6808/2830881352' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:01:00.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[40674]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:01:00.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:00.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:00:59 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:00.788 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-10T19:01:00.882 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-10T19:01:00.900 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-09-10T19:01:00.900 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-09-10T19:01:00.900 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0148416 s, 707 MB/s 2024-09-10T19:01:00.900 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-09-10T19:01:01.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:00 smithi194 ceph-mon[32464]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-10T19:01:01.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:00 smithi194 ceph-mon[32464]: osdmap e41: 7 total, 6 up, 7 in 2024-09-10T19:01:01.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:00 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:01:01.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:00 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:01:01.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:00 smithi194 ceph-mon[32464]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:01:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:00 smithi119 ceph-mon[29577]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-10T19:01:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:00 smithi119 ceph-mon[29577]: osdmap e41: 7 total, 6 up, 7 in 2024-09-10T19:01:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:00 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:01:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:00 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:01:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:00 smithi119 ceph-mon[29577]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:01:01.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:00 smithi119 ceph-mon[40674]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-10T19:01:01.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:00 smithi119 ceph-mon[40674]: osdmap e41: 7 total, 6 up, 7 in 2024-09-10T19:01:01.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:00 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:01:01.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:00 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:01:01.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:00 smithi119 ceph-mon[40674]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:01:01.863 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:01 smithi194 ceph-mon[32464]: purged_snaps scrub starts 2024-09-10T19:01:01.863 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:01 smithi194 ceph-mon[32464]: purged_snaps scrub ok 2024-09-10T19:01:01.863 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:01 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:01:01.864 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:01 smithi194 ceph-mon[32464]: osd.6 v2:172.21.15.194:6808/2830881352 boot 2024-09-10T19:01:01.864 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:01 smithi194 ceph-mon[32464]: osdmap e42: 7 total, 7 up, 7 in 2024-09-10T19:01:01.864 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:01 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:01:01.864 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:01 smithi194 ceph-mon[32464]: osdmap e43: 7 total, 7 up, 7 in 2024-09-10T19:01:02.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:01 smithi119 ceph-mon[29577]: purged_snaps scrub starts 2024-09-10T19:01:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:01 smithi119 ceph-mon[29577]: purged_snaps scrub ok 2024-09-10T19:01:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:01 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:01:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:01 smithi119 ceph-mon[29577]: osd.6 v2:172.21.15.194:6808/2830881352 boot 2024-09-10T19:01:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:01 smithi119 ceph-mon[29577]: osdmap e42: 7 total, 7 up, 7 in 2024-09-10T19:01:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:01 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:01:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:01 smithi119 ceph-mon[29577]: osdmap e43: 7 total, 7 up, 7 in 2024-09-10T19:01:02.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:01 smithi119 ceph-mon[40674]: purged_snaps scrub starts 2024-09-10T19:01:02.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:01 smithi119 ceph-mon[40674]: purged_snaps scrub ok 2024-09-10T19:01:02.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:01:02.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:01 smithi119 ceph-mon[40674]: osd.6 v2:172.21.15.194:6808/2830881352 boot 2024-09-10T19:01:02.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:01 smithi119 ceph-mon[40674]: osdmap e42: 7 total, 7 up, 7 in 2024-09-10T19:01:02.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:01:02.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:01 smithi119 ceph-mon[40674]: osdmap e43: 7 total, 7 up, 7 in 2024-09-10T19:01:02.340 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_1 2024-09-10T19:01:02.591 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:01:03.616 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:03 smithi194 ceph-mon[32464]: pgmap v145: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:03.616 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:03 smithi194 ceph-mon[32464]: osdmap e44: 7 total, 7 up, 7 in 2024-09-10T19:01:03.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:03 smithi119 ceph-mon[29577]: pgmap v145: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:03 smithi119 ceph-mon[29577]: osdmap e44: 7 total, 7 up, 7 in 2024-09-10T19:01:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:03 smithi119 ceph-mon[40674]: pgmap v145: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:03 smithi119 ceph-mon[40674]: osdmap e44: 7 total, 7 up, 7 in 2024-09-10T19:01:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.760+0000 7f30dfeb9640 1 -- 172.21.15.194:0/1667182820 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30d8066bc0 msgr2=0x7f30d80ff4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.760+0000 7f30dfeb9640 1 --2- 172.21.15.194:0/1667182820 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30d8066bc0 0x7f30d80ff4c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f30d4004950 tx=0x7f30d4009da0 comp rx=0 tx=0).stop 2024-09-10T19:01:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.760+0000 7f30dfeb9640 1 -- 172.21.15.194:0/1667182820 shutdown_connections 2024-09-10T19:01:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.760+0000 7f30dfeb9640 1 --2- 172.21.15.194:0/1667182820 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30d81024f0 0x7f30d8104970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.760+0000 7f30dfeb9640 1 --2- 172.21.15.194:0/1667182820 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f30d80ffa00 0x7f30d8101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.760+0000 7f30dfeb9640 1 --2- 172.21.15.194:0/1667182820 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30d8066bc0 0x7f30d80ff4c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.760+0000 7f30dfeb9640 1 -- 172.21.15.194:0/1667182820 >> 172.21.15.194:0/1667182820 conn(0x7f30d80f6f10 msgr2=0x7f30d80f9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.761+0000 7f30dfeb9640 1 -- 172.21.15.194:0/1667182820 shutdown_connections 2024-09-10T19:01:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.761+0000 7f30dfeb9640 1 -- 172.21.15.194:0/1667182820 wait complete. 2024-09-10T19:01:04.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.761+0000 7f30dfeb9640 1 Processor -- start 2024-09-10T19:01:04.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.761+0000 7f30dfeb9640 1 -- start start 2024-09-10T19:01:04.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.761+0000 7f30dfeb9640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30d8066bc0 0x7f30d810cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:04.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.761+0000 7f30dfeb9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30d80ffa00 0x7f30d810d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:04.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.761+0000 7f30dfeb9640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f30d81024f0 0x7f30d8114320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:04.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.761+0000 7f30dfeb9640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d81148f0 con 0x7f30d80ffa00 2024-09-10T19:01:04.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.761+0000 7f30dfeb9640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d8114a60 con 0x7f30d8066bc0 2024-09-10T19:01:04.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.761+0000 7f30dfeb9640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d8114c20 con 0x7f30d81024f0 2024-09-10T19:01:04.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30ddc55640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30d8066bc0 0x7f30d810cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:04.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30ddc55640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30d8066bc0 0x7f30d810cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:33054/0 (socket says 172.21.15.194:33054) 2024-09-10T19:01:04.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30ddc55640 1 -- 172.21.15.194:0/2591787949 learned_addr learned my addr 172.21.15.194:0/2591787949 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T19:01:04.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30de456640 1 --2- 172.21.15.194:0/2591787949 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f30d81024f0 0x7f30d8114320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:04.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30dd454640 1 --2- 172.21.15.194:0/2591787949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30d80ffa00 0x7f30d810d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:04.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30ddc55640 1 -- 172.21.15.194:0/2591787949 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f30d81024f0 msgr2=0x7f30d8114320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:04.765 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30ddc55640 1 --2- 172.21.15.194:0/2591787949 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f30d81024f0 0x7f30d8114320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30ddc55640 1 -- 172.21.15.194:0/2591787949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30d80ffa00 msgr2=0x7f30d810d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30ddc55640 1 --2- 172.21.15.194:0/2591787949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30d80ffa00 0x7f30d810d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30ddc55640 1 -- 172.21.15.194:0/2591787949 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d4005040 con 0x7f30d8066bc0 2024-09-10T19:01:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30dd454640 1 --2- 172.21.15.194:0/2591787949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30d80ffa00 0x7f30d810d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T19:01:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30ddc55640 1 --2- 172.21.15.194:0/2591787949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30d8066bc0 0x7f30d810cdf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009400 tx=0x7f30d4009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30ce7fc640 1 -- 172.21.15.194:0/2591787949 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d402abf0 con 0x7f30d8066bc0 2024-09-10T19:01:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30ce7fc640 1 -- 172.21.15.194:0/2591787949 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f30d402ad50 con 0x7f30d8066bc0 2024-09-10T19:01:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30dfeb9640 1 -- 172.21.15.194:0/2591787949 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d8114ea0 con 0x7f30d8066bc0 2024-09-10T19:01:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30ce7fc640 1 -- 172.21.15.194:0/2591787949 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d4033730 con 0x7f30d8066bc0 2024-09-10T19:01:04.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.762+0000 7f30dfeb9640 1 -- 172.21.15.194:0/2591787949 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d81152e0 con 0x7f30d8066bc0 2024-09-10T19:01:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.763+0000 7f30dfeb9640 1 -- 172.21.15.194:0/2591787949 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d804bf60 con 0x7f30d8066bc0 2024-09-10T19:01:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.764+0000 7f30ce7fc640 1 -- 172.21.15.194:0/2591787949 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 15) v1 ==== 90746+0+0 (secure 0 0 0) 0x7f30d4030020 con 0x7f30d8066bc0 2024-09-10T19:01:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.764+0000 7f30ce7fc640 1 --2- 172.21.15.194:0/2591787949 >> v2:172.21.15.119:6800/1094129222 conn(0x7f30c406ee10 0x7f30c40712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.764+0000 7f30ce7fc640 1 -- 172.21.15.194:0/2591787949 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 3785+0+0 (secure 0 0 0) 0x7f30d40ad3e0 con 0x7f30d8066bc0 2024-09-10T19:01:04.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.765+0000 7f30dd454640 1 --2- 172.21.15.194:0/2591787949 >> v2:172.21.15.119:6800/1094129222 conn(0x7f30c406ee10 0x7f30c40712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:04.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.765+0000 7f30dd454640 1 --2- 172.21.15.194:0/2591787949 >> v2:172.21.15.119:6800/1094129222 conn(0x7f30c406ee10 0x7f30c40712c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f30d810e280 tx=0x7f30c800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:04.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.768+0000 7f30ce7fc640 1 -- 172.21.15.194:0/2591787949 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f30d407acd0 con 0x7f30d8066bc0 2024-09-10T19:01:04.936 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:04.934+0000 7f30dfeb9640 1 -- 172.21.15.194:0/2591787949 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f30d8107850 con 0x7f30c406ee10 2024-09-10T19:01:05.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: Detected new or changed devices on smithi194 2024-09-10T19:01:05.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:05.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:05.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:05.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: Adjusting osd_memory_target on smithi194 to 5696M 2024-09-10T19:01:05.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:05.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:05.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:05.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:05.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:05.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:04 smithi194 ceph-mon[32464]: pgmap v147: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:05.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: Detected new or changed devices on smithi194 2024-09-10T19:01:05.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:05.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:05.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:05.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: Adjusting osd_memory_target on smithi194 to 5696M 2024-09-10T19:01:05.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:05.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:05.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:05.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:05.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:05.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[29577]: pgmap v147: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:05.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: Detected new or changed devices on smithi194 2024-09-10T19:01:05.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:05.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:05.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:05.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: Adjusting osd_memory_target on smithi194 to 5696M 2024-09-10T19:01:05.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:05.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:05.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:05.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:05.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:05.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:05.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:04 smithi119 ceph-mon[40674]: pgmap v147: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:06.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:05 smithi194 ceph-mon[32464]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:06.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:05 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:01:06.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:05 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:01:06.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:05 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:06.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:05 smithi119 ceph-mon[29577]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:06.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:05 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:01:06.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:05 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:01:06.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:05 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:06.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:05 smithi119 ceph-mon[40674]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:06.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:05 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:01:06.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:05 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:01:06.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:05 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:07.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:06 smithi194 ceph-mon[32464]: pgmap v148: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:07.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:06 smithi119 ceph-mon[29577]: pgmap v148: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:07.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:06 smithi119 ceph-mon[40674]: pgmap v148: 1 pgs: 1 peering; 0 B data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/246732933' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16049fee-bcde-450e-b4c5-de13dbb392d4"}]: dispatch 2024-09-10T19:01:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16049fee-bcde-450e-b4c5-de13dbb392d4"}]: dispatch 2024-09-10T19:01:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16049fee-bcde-450e-b4c5-de13dbb392d4"}]': finished 2024-09-10T19:01:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:07 smithi194 ceph-mon[32464]: osdmap e45: 8 total, 7 up, 8 in 2024-09-10T19:01:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:07 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/2039693557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:01:08.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/246732933' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16049fee-bcde-450e-b4c5-de13dbb392d4"}]: dispatch 2024-09-10T19:01:08.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16049fee-bcde-450e-b4c5-de13dbb392d4"}]: dispatch 2024-09-10T19:01:08.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16049fee-bcde-450e-b4c5-de13dbb392d4"}]': finished 2024-09-10T19:01:08.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:07 smithi119 ceph-mon[29577]: osdmap e45: 8 total, 7 up, 8 in 2024-09-10T19:01:08.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:07 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:08.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/2039693557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:01:08.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/246732933' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16049fee-bcde-450e-b4c5-de13dbb392d4"}]: dispatch 2024-09-10T19:01:08.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16049fee-bcde-450e-b4c5-de13dbb392d4"}]: dispatch 2024-09-10T19:01:08.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16049fee-bcde-450e-b4c5-de13dbb392d4"}]': finished 2024-09-10T19:01:08.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:07 smithi119 ceph-mon[40674]: osdmap e45: 8 total, 7 up, 8 in 2024-09-10T19:01:08.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:07 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:08.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/2039693557' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:01:09.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:08 smithi194 ceph-mon[32464]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:09.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:08 smithi119 ceph-mon[29577]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:09.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:08 smithi119 ceph-mon[40674]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:11.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:11 smithi194 ceph-mon[32464]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:11.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:11 smithi119 ceph-mon[29577]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:11.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:11 smithi119 ceph-mon[40674]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:13.467 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:13 smithi194 ceph-mon[32464]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:13.467 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:13 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-10T19:01:13.467 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:13 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:13.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:13 smithi119 ceph-mon[29577]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:13.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:13 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-10T19:01:13.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:13 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:13.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:13 smithi119 ceph-mon[40674]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:13.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:13 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-10T19:01:13.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:13 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:14.494 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:14 smithi194 ceph-mon[32464]: Deploying daemon osd.7 on smithi194 2024-09-10T19:01:14.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:14 smithi119 ceph-mon[29577]: Deploying daemon osd.7 on smithi194 2024-09-10T19:01:14.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:14 smithi119 ceph-mon[40674]: Deploying daemon osd.7 on smithi194 2024-09-10T19:01:15.334 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:15 smithi194 ceph-mon[32464]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:15.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:15 smithi119 ceph-mon[29577]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:15 smithi119 ceph-mon[40674]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:17.441 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:17 smithi194 ceph-mon[32464]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:17.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:17 smithi119 ceph-mon[29577]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:17.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:17 smithi119 ceph-mon[40674]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:18.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:18 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:18.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:18 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:18.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:18 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:18.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:18 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:18.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:18 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:18.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:18 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:18.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:18 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:19.441 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:19 smithi194 ceph-mon[32464]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:19.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:19 smithi119 ceph-mon[29577]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:19.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:19 smithi119 ceph-mon[40674]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:21.141 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 7 on host 'smithi194' 2024-09-10T19:01:21.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:21.139+0000 7f30ce7fc640 1 -- 172.21.15.194:0/2591787949 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f30d8107850 con 0x7f30c406ee10 2024-09-10T19:01:21.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:21.143+0000 7f30dfeb9640 1 -- 172.21.15.194:0/2591787949 >> v2:172.21.15.119:6800/1094129222 conn(0x7f30c406ee10 msgr2=0x7f30c40712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:21.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:21.143+0000 7f30dfeb9640 1 --2- 172.21.15.194:0/2591787949 >> v2:172.21.15.119:6800/1094129222 conn(0x7f30c406ee10 0x7f30c40712c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f30d810e280 tx=0x7f30c800b040 comp rx=0 tx=0).stop 2024-09-10T19:01:21.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:21.143+0000 7f30dfeb9640 1 -- 172.21.15.194:0/2591787949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30d8066bc0 msgr2=0x7f30d810cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:21.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:21.143+0000 7f30dfeb9640 1 --2- 172.21.15.194:0/2591787949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30d8066bc0 0x7f30d810cdf0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009400 tx=0x7f30d4009da0 comp rx=0 tx=0).stop 2024-09-10T19:01:21.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:21.143+0000 7f30dfeb9640 1 -- 172.21.15.194:0/2591787949 shutdown_connections 2024-09-10T19:01:21.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:21.143+0000 7f30dfeb9640 1 --2- 172.21.15.194:0/2591787949 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f30d81024f0 0x7f30d8114320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:21.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:21.143+0000 7f30dfeb9640 1 --2- 172.21.15.194:0/2591787949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30d80ffa00 0x7f30d810d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:21.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:21.143+0000 7f30dfeb9640 1 --2- 172.21.15.194:0/2591787949 >> v2:172.21.15.119:6800/1094129222 conn(0x7f30c406ee10 0x7f30c40712c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:21.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:21.143+0000 7f30dfeb9640 1 --2- 172.21.15.194:0/2591787949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30d8066bc0 0x7f30d810cdf0 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009400 tx=0x7f30d4009da0 comp rx=0 tx=0).stop 2024-09-10T19:01:21.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:21.143+0000 7f30dfeb9640 1 -- 172.21.15.194:0/2591787949 >> 172.21.15.194:0/2591787949 conn(0x7f30d80f6f10 msgr2=0x7f30d80fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:21.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:21.144+0000 7f30dfeb9640 1 -- 172.21.15.194:0/2591787949 shutdown_connections 2024-09-10T19:01:21.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:21.144+0000 7f30dfeb9640 1 -- 172.21.15.194:0/2591787949 wait complete. 2024-09-10T19:01:21.445 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:21 smithi194 ceph-mon[32464]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:21.445 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:21 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:21.445 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:21 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:21.445 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:21 smithi194 ceph-mon[32464]: from='osd.7 v2:172.21.15.194:6812/117762113' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T19:01:21.445 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:21 smithi194 ceph-mon[32464]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T19:01:21.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:21 smithi119 ceph-mon[29577]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:21.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:21 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:21.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:21 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:21.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:21 smithi119 ceph-mon[29577]: from='osd.7 v2:172.21.15.194:6812/117762113' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T19:01:21.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:21 smithi119 ceph-mon[29577]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T19:01:21.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:21 smithi119 ceph-mon[40674]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:21.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:21 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:21.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:21 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:21.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:21 smithi119 ceph-mon[40674]: from='osd.7 v2:172.21.15.194:6812/117762113' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T19:01:21.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:21 smithi119 ceph-mon[40674]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T19:01:21.764 DEBUG:teuthology.orchestra.run.smithi194:osd.7> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.7.service 2024-09-10T19:01:21.766 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-10T19:01:21.766 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-10T19:01:22.022 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:23.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:22 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:23.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:22 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:23.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:22 smithi194 ceph-mon[32464]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-10T19:01:23.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:22 smithi194 ceph-mon[32464]: osdmap e46: 8 total, 7 up, 8 in 2024-09-10T19:01:23.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:22 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:23.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:22 smithi194 ceph-mon[32464]: from='osd.7 v2:172.21.15.194:6812/117762113' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:01:23.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:22 smithi194 ceph-mon[32464]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:01:23.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:22 smithi194 ceph-mon[32464]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:23.269 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:01:23 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:01:23.147+0000 7f7778986640 -1 osd.7 0 waiting for initial osdmap 2024-09-10T19:01:23.269 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:01:23 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:01:23.153+0000 7f7772fbc640 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:01:23.301 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:23.301 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:23.301 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[29577]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-10T19:01:23.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[29577]: osdmap e46: 8 total, 7 up, 8 in 2024-09-10T19:01:23.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:23.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[29577]: from='osd.7 v2:172.21.15.194:6812/117762113' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:01:23.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[29577]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:01:23.302 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[29577]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:23.302 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:23.302 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:23.302 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[40674]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-10T19:01:23.302 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[40674]: osdmap e46: 8 total, 7 up, 8 in 2024-09-10T19:01:23.302 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:23.302 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[40674]: from='osd.7 v2:172.21.15.194:6812/117762113' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:01:23.303 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[40674]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-09-10T19:01:23.303 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:22 smithi119 ceph-mon[40674]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:01:23.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.812+0000 7f4f30479640 1 -- 172.21.15.119:0/2567010690 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f280fb760 msgr2=0x7f4f280fbb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:23.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.812+0000 7f4f30479640 1 --2- 172.21.15.119:0/2567010690 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f280fb760 0x7f4f280fbb70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4f24004890 tx=0x7f4f24009da0 comp rx=0 tx=0).stop 2024-09-10T19:01:23.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 -- 172.21.15.119:0/2567010690 shutdown_connections 2024-09-10T19:01:23.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 --2- 172.21.15.119:0/2567010690 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f280fd3e0 0x7f4f28103c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:23.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 --2- 172.21.15.119:0/2567010690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f280fca30 0x7f4f280fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:23.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 --2- 172.21.15.119:0/2567010690 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f280fb760 0x7f4f280fbb70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:23.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 -- 172.21.15.119:0/2567010690 >> 172.21.15.119:0/2567010690 conn(0x7f4f280f6f20 msgr2=0x7f4f280f9350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 -- 172.21.15.119:0/2567010690 shutdown_connections 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 -- 172.21.15.119:0/2567010690 wait complete. 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 Processor -- start 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 -- start start 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f280fb760 0x7f4f280669e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f280fca30 0x7f4f28111b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f280fd3e0 0x7f4f281140f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f281148d0 con 0x7f4f280fd3e0 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f28114a40 con 0x7f4f280fca30 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f30479640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f28114c00 con 0x7f4f280fb760 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f2e215640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f280fb760 0x7f4f280669e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f2ea16640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f280fd3e0 0x7f4f281140f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f2e215640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f280fb760 0x7f4f280669e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:53750/0 (socket says 172.21.15.119:53750) 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.813+0000 7f4f2e215640 1 -- 172.21.15.119:0/394892223 learned_addr learned my addr 172.21.15.119:0/394892223 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:01:23.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.814+0000 7f4f2da14640 1 --2- 172.21.15.119:0/394892223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f280fca30 0x7f4f28111b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:23.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.814+0000 7f4f2ea16640 1 -- 172.21.15.119:0/394892223 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f280fb760 msgr2=0x7f4f280669e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:23.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.814+0000 7f4f2ea16640 1 --2- 172.21.15.119:0/394892223 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f280fb760 0x7f4f280669e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:23.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.814+0000 7f4f2ea16640 1 -- 172.21.15.119:0/394892223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f280fca30 msgr2=0x7f4f28111b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:23.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.814+0000 7f4f2ea16640 1 --2- 172.21.15.119:0/394892223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f280fca30 0x7f4f28111b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:23.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.814+0000 7f4f2ea16640 1 -- 172.21.15.119:0/394892223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f200098b0 con 0x7f4f280fd3e0 2024-09-10T19:01:23.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.814+0000 7f4f2e215640 1 --2- 172.21.15.119:0/394892223 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f280fb760 0x7f4f280669e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:01:23.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.814+0000 7f4f2ea16640 1 --2- 172.21.15.119:0/394892223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f280fd3e0 0x7f4f281140f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4f20001e80 tx=0x7f4f20004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:23.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.814+0000 7f4f1effd640 1 -- 172.21.15.119:0/394892223 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2000fdd0 con 0x7f4f280fd3e0 2024-09-10T19:01:23.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.814+0000 7f4f1effd640 1 -- 172.21.15.119:0/394892223 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f4f20007d00 con 0x7f4f280fd3e0 2024-09-10T19:01:23.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.814+0000 7f4f30479640 1 -- 172.21.15.119:0/394892223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f24005040 con 0x7f4f280fd3e0 2024-09-10T19:01:23.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.814+0000 7f4f1effd640 1 -- 172.21.15.119:0/394892223 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f20015470 con 0x7f4f280fd3e0 2024-09-10T19:01:23.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.814+0000 7f4f30479640 1 -- 172.21.15.119:0/394892223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f28115170 con 0x7f4f280fd3e0 2024-09-10T19:01:23.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.815+0000 7f4f30479640 1 -- 172.21.15.119:0/394892223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f2804bf60 con 0x7f4f280fd3e0 2024-09-10T19:01:23.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.816+0000 7f4f1effd640 1 -- 172.21.15.119:0/394892223 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 15) v1 ==== 90746+0+0 (secure 0 0 0) 0x7f4f20007890 con 0x7f4f280fd3e0 2024-09-10T19:01:23.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.816+0000 7f4f1effd640 1 --2- 172.21.15.119:0/394892223 >> v2:172.21.15.119:6800/1094129222 conn(0x7f4f1406ede0 0x7f4f14071290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:23.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.816+0000 7f4f2e215640 1 --2- 172.21.15.119:0/394892223 >> v2:172.21.15.119:6800/1094129222 conn(0x7f4f1406ede0 0x7f4f14071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:23.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.816+0000 7f4f1effd640 1 -- 172.21.15.119:0/394892223 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 3937+0+0 (secure 0 0 0) 0x7f4f2008df40 con 0x7f4f280fd3e0 2024-09-10T19:01:23.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.817+0000 7f4f2e215640 1 --2- 172.21.15.119:0/394892223 >> v2:172.21.15.119:6800/1094129222 conn(0x7f4f1406ede0 0x7f4f14071290 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4f240040d0 tx=0x7f4f24009da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:23.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.820+0000 7f4f1effd640 1 -- 172.21.15.119:0/394892223 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f4f2005b700 con 0x7f4f280fd3e0 2024-09-10T19:01:23.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.976+0000 7f4f30479640 1 -- 172.21.15.119:0/394892223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f4f28106710 con 0x7f4f280fd3e0 2024-09-10T19:01:23.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.977+0000 7f4f1effd640 1 -- 172.21.15.119:0/394892223 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v47) v1 ==== 74+0+130 (secure 0 0 0) 0x7f4f2001a0a0 con 0x7f4f280fd3e0 2024-09-10T19:01:23.979 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T19:01:23.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.979+0000 7f4f30479640 1 -- 172.21.15.119:0/394892223 >> v2:172.21.15.119:6800/1094129222 conn(0x7f4f1406ede0 msgr2=0x7f4f14071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:23.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.979+0000 7f4f30479640 1 --2- 172.21.15.119:0/394892223 >> v2:172.21.15.119:6800/1094129222 conn(0x7f4f1406ede0 0x7f4f14071290 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4f240040d0 tx=0x7f4f24009da0 comp rx=0 tx=0).stop 2024-09-10T19:01:23.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.979+0000 7f4f30479640 1 -- 172.21.15.119:0/394892223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f280fd3e0 msgr2=0x7f4f281140f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:23.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.979+0000 7f4f30479640 1 --2- 172.21.15.119:0/394892223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f280fd3e0 0x7f4f281140f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4f20001e80 tx=0x7f4f20004d10 comp rx=0 tx=0).stop 2024-09-10T19:01:23.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.979+0000 7f4f30479640 1 -- 172.21.15.119:0/394892223 shutdown_connections 2024-09-10T19:01:23.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.979+0000 7f4f30479640 1 --2- 172.21.15.119:0/394892223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4f280fd3e0 0x7f4f281140f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:23.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.979+0000 7f4f30479640 1 --2- 172.21.15.119:0/394892223 >> v2:172.21.15.119:6800/1094129222 conn(0x7f4f1406ede0 0x7f4f14071290 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:23.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.979+0000 7f4f30479640 1 --2- 172.21.15.119:0/394892223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4f280fca30 0x7f4f28111b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:23.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.979+0000 7f4f30479640 1 --2- 172.21.15.119:0/394892223 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4f280fb760 0x7f4f280669e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:23.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.979+0000 7f4f30479640 1 -- 172.21.15.119:0/394892223 >> 172.21.15.119:0/394892223 conn(0x7f4f280f6f20 msgr2=0x7f4f280ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:23.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.979+0000 7f4f30479640 1 -- 172.21.15.119:0/394892223 shutdown_connections 2024-09-10T19:01:23.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:23.979+0000 7f4f30479640 1 -- 172.21.15.119:0/394892223 wait complete. 2024-09-10T19:01:24.263 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:24 smithi194 ceph-mon[32464]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-10T19:01:24.263 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:24 smithi194 ceph-mon[32464]: osdmap e47: 8 total, 7 up, 8 in 2024-09-10T19:01:24.263 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:24 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:24.263 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:24 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:24.264 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/394892223' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T19:01:24.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:24 smithi119 ceph-mon[40674]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-10T19:01:24.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:24 smithi119 ceph-mon[40674]: osdmap e47: 8 total, 7 up, 8 in 2024-09-10T19:01:24.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:24 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:24.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:24 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:24.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/394892223' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T19:01:24.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:24 smithi119 ceph-mon[29577]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-09-10T19:01:24.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:24 smithi119 ceph-mon[29577]: osdmap e47: 8 total, 7 up, 8 in 2024-09-10T19:01:24.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:24 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:24.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:24 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:24.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/394892223' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T19:01:24.816 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":47,"num_osds":8,"num_up_osds":7,"osd_up_since":1725994860,"num_in_osds":8,"osd_in_since":1725994867,"num_remapped_pgs":0} 2024-09-10T19:01:25.250 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:25 smithi194 ceph-mon[32464]: purged_snaps scrub starts 2024-09-10T19:01:25.250 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:25 smithi194 ceph-mon[32464]: purged_snaps scrub ok 2024-09-10T19:01:25.250 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:25 smithi194 ceph-mon[32464]: osd.7 v2:172.21.15.194:6812/117762113 boot 2024-09-10T19:01:25.250 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:25 smithi194 ceph-mon[32464]: osdmap e48: 8 total, 8 up, 8 in 2024-09-10T19:01:25.250 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:25 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:25.250 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:25 smithi194 ceph-mon[32464]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:25.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:25 smithi119 ceph-mon[29577]: purged_snaps scrub starts 2024-09-10T19:01:25.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:25 smithi119 ceph-mon[29577]: purged_snaps scrub ok 2024-09-10T19:01:25.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:25 smithi119 ceph-mon[29577]: osd.7 v2:172.21.15.194:6812/117762113 boot 2024-09-10T19:01:25.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:25 smithi119 ceph-mon[29577]: osdmap e48: 8 total, 8 up, 8 in 2024-09-10T19:01:25.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:25 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:25.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:25 smithi119 ceph-mon[29577]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:25.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:25 smithi119 ceph-mon[40674]: purged_snaps scrub starts 2024-09-10T19:01:25.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:25 smithi119 ceph-mon[40674]: purged_snaps scrub ok 2024-09-10T19:01:25.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:25 smithi119 ceph-mon[40674]: osd.7 v2:172.21.15.194:6812/117762113 boot 2024-09-10T19:01:25.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:25 smithi119 ceph-mon[40674]: osdmap e48: 8 total, 8 up, 8 in 2024-09-10T19:01:25.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:25 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:01:25.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:25 smithi119 ceph-mon[40674]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:25.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-10T19:01:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:26.471 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: osdmap e49: 8 total, 8 up, 8 in 2024-09-10T19:01:26.471 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: Detected new or changed devices on smithi194 2024-09-10T19:01:26.471 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:26.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:26.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:26.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:26.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:26.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:26.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: Adjusting osd_memory_target on smithi194 to 4272M 2024-09-10T19:01:26.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:26.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:26.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:26.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:26.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:26.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:26.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:26.473 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:26.473 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: osdmap e49: 8 total, 8 up, 8 in 2024-09-10T19:01:26.473 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: Detected new or changed devices on smithi194 2024-09-10T19:01:26.473 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:26.473 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:26.473 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:26.473 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:26.473 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:26.473 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:26.473 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: Adjusting osd_memory_target on smithi194 to 4272M 2024-09-10T19:01:26.474 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:26.474 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:26.474 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:26.474 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:26.474 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:26.474 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:26.474 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:26.474 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:26 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:26.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: osdmap e49: 8 total, 8 up, 8 in 2024-09-10T19:01:26.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: Detected new or changed devices on smithi194 2024-09-10T19:01:26.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:01:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: Adjusting osd_memory_target on smithi194 to 4272M 2024-09-10T19:01:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:26.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:26 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:27.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:27 smithi119 ceph-mon[29577]: osdmap e50: 8 total, 8 up, 8 in 2024-09-10T19:01:27.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:27 smithi119 ceph-mon[29577]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:27.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:27 smithi119 ceph-mon[40674]: osdmap e50: 8 total, 8 up, 8 in 2024-09-10T19:01:27.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:27 smithi119 ceph-mon[40674]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:27.419 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:27 smithi194 ceph-mon[32464]: osdmap e50: 8 total, 8 up, 8 in 2024-09-10T19:01:27.419 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:27 smithi194 ceph-mon[32464]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:27.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.880+0000 7f9872c2d640 1 -- 172.21.15.119:0/3590608581 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f986c0fd3c0 msgr2=0x7f986c103c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.880+0000 7f9872c2d640 1 --2- 172.21.15.119:0/3590608581 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f986c0fd3c0 0x7f986c103c60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9864009d20 tx=0x7f9864004c50 comp rx=0 tx=0).stop 2024-09-10T19:01:27.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.880+0000 7f9872c2d640 1 -- 172.21.15.119:0/3590608581 shutdown_connections 2024-09-10T19:01:27.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.880+0000 7f9872c2d640 1 --2- 172.21.15.119:0/3590608581 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f986c0fd3c0 0x7f986c103c60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:27.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.880+0000 7f9872c2d640 1 --2- 172.21.15.119:0/3590608581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f986c0fca10 0x7f986c0fce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:27.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.880+0000 7f9872c2d640 1 --2- 172.21.15.119:0/3590608581 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f986c0fb740 0x7f986c0fbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:27.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.880+0000 7f9872c2d640 1 -- 172.21.15.119:0/3590608581 >> 172.21.15.119:0/3590608581 conn(0x7f986c0f6ee0 msgr2=0x7f986c0f9330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:27.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.880+0000 7f9872c2d640 1 -- 172.21.15.119:0/3590608581 shutdown_connections 2024-09-10T19:01:27.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.880+0000 7f9872c2d640 1 -- 172.21.15.119:0/3590608581 wait complete. 2024-09-10T19:01:27.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.880+0000 7f9872c2d640 1 Processor -- start 2024-09-10T19:01:27.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.880+0000 7f9872c2d640 1 -- start start 2024-09-10T19:01:27.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f9872c2d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f986c0fb740 0x7f986c0669e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:27.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f9872c2d640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f986c0fca10 0x7f986c111840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:27.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f9872c2d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f986c0fd3c0 0x7f986c113e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:27.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f9872c2d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986c1145b0 con 0x7f986c0fb740 2024-09-10T19:01:27.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f9872c2d640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986c114720 con 0x7f986c0fca10 2024-09-10T19:01:27.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f9872c2d640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f986c1148e0 con 0x7f986c0fd3c0 2024-09-10T19:01:27.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f98709c9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f986c0fb740 0x7f986c0669e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:27.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f98711ca640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f986c0fd3c0 0x7f986c113e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:27.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f98709c9640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f986c0fb740 0x7f986c0669e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58956/0 (socket says 172.21.15.119:58956) 2024-09-10T19:01:27.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f98709c9640 1 -- 172.21.15.119:0/834675845 learned_addr learned my addr 172.21.15.119:0/834675845 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:01:27.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f98711ca640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f986c0fd3c0 0x7f986c113e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:53762/0 (socket says 172.21.15.119:53762) 2024-09-10T19:01:27.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f986bfff640 1 --2- 172.21.15.119:0/834675845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f986c0fca10 0x7f986c111840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:27.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f98711ca640 1 -- 172.21.15.119:0/834675845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f986c0fca10 msgr2=0x7f986c111840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f98711ca640 1 --2- 172.21.15.119:0/834675845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f986c0fca10 0x7f986c111840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:27.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f98711ca640 1 -- 172.21.15.119:0/834675845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f986c0fb740 msgr2=0x7f986c0669e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:27.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f98711ca640 1 --2- 172.21.15.119:0/834675845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f986c0fb740 0x7f986c0669e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:27.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f98711ca640 1 -- 172.21.15.119:0/834675845 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9860005040 con 0x7f986c0fd3c0 2024-09-10T19:01:27.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f98711ca640 1 --2- 172.21.15.119:0/834675845 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f986c0fd3c0 0x7f986c113e20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f986402a740 tx=0x7f9864004c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:27.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f98697fa640 1 -- 172.21.15.119:0/834675845 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9864038070 con 0x7f986c0fd3c0 2024-09-10T19:01:27.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f98697fa640 1 -- 172.21.15.119:0/834675845 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f986403c410 con 0x7f986c0fd3c0 2024-09-10T19:01:27.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f9872c2d640 1 -- 172.21.15.119:0/834675845 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98640098b0 con 0x7f986c0fd3c0 2024-09-10T19:01:27.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.881+0000 7f98697fa640 1 -- 172.21.15.119:0/834675845 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986403b370 con 0x7f986c0fd3c0 2024-09-10T19:01:27.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.882+0000 7f9872c2d640 1 -- 172.21.15.119:0/834675845 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f986c114ef0 con 0x7f986c0fd3c0 2024-09-10T19:01:27.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.883+0000 7f9872c2d640 1 -- 172.21.15.119:0/834675845 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f986c04bf60 con 0x7f986c0fd3c0 2024-09-10T19:01:27.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.883+0000 7f98697fa640 1 -- 172.21.15.119:0/834675845 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 15) v1 ==== 90746+0+0 (secure 0 0 0) 0x7f9864030070 con 0x7f986c0fd3c0 2024-09-10T19:01:27.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.883+0000 7f98697fa640 1 --2- 172.21.15.119:0/834675845 >> v2:172.21.15.119:6800/1094129222 conn(0x7f985406ede0 0x7f9854071290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:27.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.883+0000 7f98697fa640 1 -- 172.21.15.119:0/834675845 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4077+0+0 (secure 0 0 0) 0x7f98640ae470 con 0x7f986c0fd3c0 2024-09-10T19:01:27.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.883+0000 7f98709c9640 1 --2- 172.21.15.119:0/834675845 >> v2:172.21.15.119:6800/1094129222 conn(0x7f985406ede0 0x7f9854071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:27.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.884+0000 7f98709c9640 1 --2- 172.21.15.119:0/834675845 >> v2:172.21.15.119:6800/1094129222 conn(0x7f985406ede0 0x7f9854071290 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f985c0041d0 tx=0x7f985c009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:27.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:27.887+0000 7f98697fa640 1 -- 172.21.15.119:0/834675845 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f98640762a0 con 0x7f986c0fd3c0 2024-09-10T19:01:28.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:28.045+0000 7f9872c2d640 1 -- 172.21.15.119:0/834675845 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f986c02aa40 con 0x7f986c0fd3c0 2024-09-10T19:01:28.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:28.045+0000 7f98697fa640 1 -- 172.21.15.119:0/834675845 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v50) v1 ==== 74+0+130 (secure 0 0 0) 0x7f98640410a0 con 0x7f986c0fd3c0 2024-09-10T19:01:28.047 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T19:01:28.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:28.048+0000 7f9872c2d640 1 -- 172.21.15.119:0/834675845 >> v2:172.21.15.119:6800/1094129222 conn(0x7f985406ede0 msgr2=0x7f9854071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:28.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:28.048+0000 7f9872c2d640 1 --2- 172.21.15.119:0/834675845 >> v2:172.21.15.119:6800/1094129222 conn(0x7f985406ede0 0x7f9854071290 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f985c0041d0 tx=0x7f985c009040 comp rx=0 tx=0).stop 2024-09-10T19:01:28.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:28.048+0000 7f9872c2d640 1 -- 172.21.15.119:0/834675845 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f986c0fd3c0 msgr2=0x7f986c113e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:28.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:28.048+0000 7f9872c2d640 1 --2- 172.21.15.119:0/834675845 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f986c0fd3c0 0x7f986c113e20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f986402a740 tx=0x7f9864004c50 comp rx=0 tx=0).stop 2024-09-10T19:01:28.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:28.048+0000 7f9872c2d640 1 -- 172.21.15.119:0/834675845 shutdown_connections 2024-09-10T19:01:28.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:28.048+0000 7f9872c2d640 1 --2- 172.21.15.119:0/834675845 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f986c0fd3c0 0x7f986c113e20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:28.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:28.048+0000 7f9872c2d640 1 --2- 172.21.15.119:0/834675845 >> v2:172.21.15.119:6800/1094129222 conn(0x7f985406ede0 0x7f9854071290 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:28.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:28.048+0000 7f9872c2d640 1 --2- 172.21.15.119:0/834675845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f986c0fca10 0x7f986c111840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:28.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:28.048+0000 7f9872c2d640 1 --2- 172.21.15.119:0/834675845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f986c0fb740 0x7f986c0669e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:28.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:28.048+0000 7f9872c2d640 1 -- 172.21.15.119:0/834675845 >> 172.21.15.119:0/834675845 conn(0x7f986c0f6ee0 msgr2=0x7f986c0ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:28.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:28.048+0000 7f9872c2d640 1 -- 172.21.15.119:0/834675845 shutdown_connections 2024-09-10T19:01:28.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:28.048+0000 7f9872c2d640 1 -- 172.21.15.119:0/834675845 wait complete. 2024-09-10T19:01:28.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/834675845' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T19:01:28.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/834675845' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T19:01:28.436 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/834675845' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T19:01:28.613 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":50,"num_osds":8,"num_up_osds":8,"osd_up_since":1725994884,"num_in_osds":8,"osd_in_since":1725994867,"num_remapped_pgs":0} 2024-09-10T19:01:28.613 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T19:01:28.859 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:29.268 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:29 smithi119 ceph-mon[29577]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:29.268 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:29 smithi119 ceph-mon[40674]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:29.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:29 smithi194 ceph-mon[32464]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:30.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.725+0000 7f98a5343640 1 -- 172.21.15.119:0/3007738898 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f98a00fb950 msgr2=0x7f98a00fbdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:30.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.725+0000 7f98a5343640 1 --2- 172.21.15.119:0/3007738898 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f98a00fb950 0x7f98a00fbdc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9888004980 tx=0x7f9888009da0 comp rx=0 tx=0).stop 2024-09-10T19:01:30.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.725+0000 7f98a5343640 1 -- 172.21.15.119:0/3007738898 shutdown_connections 2024-09-10T19:01:30.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.725+0000 7f98a5343640 1 --2- 172.21.15.119:0/3007738898 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98a00fc300 0x7f98a0108560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:30.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.725+0000 7f98a5343640 1 --2- 172.21.15.119:0/3007738898 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f98a00fb950 0x7f98a00fbdc0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:30.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.725+0000 7f98a5343640 1 --2- 172.21.15.119:0/3007738898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98a01017b0 0x7f98a0101b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:30.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.725+0000 7f98a5343640 1 -- 172.21.15.119:0/3007738898 >> 172.21.15.119:0/3007738898 conn(0x7f98a00724d0 msgr2=0x7f98a00728d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:30.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.725+0000 7f98a5343640 1 -- 172.21.15.119:0/3007738898 shutdown_connections 2024-09-10T19:01:30.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.725+0000 7f98a5343640 1 -- 172.21.15.119:0/3007738898 wait complete. 2024-09-10T19:01:30.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98a5343640 1 Processor -- start 2024-09-10T19:01:30.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98a5343640 1 -- start start 2024-09-10T19:01:30.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98a5343640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98a00fb950 0x7f98a0112e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:30.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98a5343640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f98a00fc300 0x7f98a0113390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:30.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98a5343640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98a01017b0 0x7f98a010cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:30.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98a5343640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a010d630 con 0x7f98a01017b0 2024-09-10T19:01:30.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98a5343640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a010d7a0 con 0x7f98a00fb950 2024-09-10T19:01:30.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98a5343640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a010d960 con 0x7f98a00fc300 2024-09-10T19:01:30.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98967fc640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f98a00fc300 0x7f98a0113390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:30.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98967fc640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f98a00fc300 0x7f98a0113390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:53784/0 (socket says 172.21.15.119:53784) 2024-09-10T19:01:30.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98967fc640 1 -- 172.21.15.119:0/1696867784 learned_addr learned my addr 172.21.15.119:0/1696867784 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:01:30.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f989f7fe640 1 --2- 172.21.15.119:0/1696867784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98a01017b0 0x7f98a010cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:30.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f989effd640 1 --2- 172.21.15.119:0/1696867784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98a00fb950 0x7f98a0112e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:30.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98967fc640 1 -- 172.21.15.119:0/1696867784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98a00fb950 msgr2=0x7f98a0112e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:30.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98967fc640 1 --2- 172.21.15.119:0/1696867784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98a00fb950 0x7f98a0112e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:30.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98967fc640 1 -- 172.21.15.119:0/1696867784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98a01017b0 msgr2=0x7f98a010cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:30.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98967fc640 1 --2- 172.21.15.119:0/1696867784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98a01017b0 0x7f98a010cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:30.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.726+0000 7f98967fc640 1 -- 172.21.15.119:0/1696867784 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98980098b0 con 0x7f98a00fc300 2024-09-10T19:01:30.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.727+0000 7f98967fc640 1 --2- 172.21.15.119:0/1696867784 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f98a00fc300 0x7f98a0113390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f98880094b0 tx=0x7f9888009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:30.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.727+0000 7f9897fff640 1 -- 172.21.15.119:0/1696867784 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988802bcb0 con 0x7f98a00fc300 2024-09-10T19:01:30.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.727+0000 7f9897fff640 1 -- 172.21.15.119:0/1696867784 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f988802be10 con 0x7f98a00fc300 2024-09-10T19:01:30.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.727+0000 7f98a5343640 1 -- 172.21.15.119:0/1696867784 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9888005040 con 0x7f98a00fc300 2024-09-10T19:01:30.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.727+0000 7f9897fff640 1 -- 172.21.15.119:0/1696867784 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98880336b0 con 0x7f98a00fc300 2024-09-10T19:01:30.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.727+0000 7f98a5343640 1 -- 172.21.15.119:0/1696867784 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a0119ca0 con 0x7f98a00fc300 2024-09-10T19:01:30.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.728+0000 7f98a5343640 1 -- 172.21.15.119:0/1696867784 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a00fced0 con 0x7f98a00fc300 2024-09-10T19:01:30.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.728+0000 7f9897fff640 1 -- 172.21.15.119:0/1696867784 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 15) v1 ==== 90746+0+0 (secure 0 0 0) 0x7f9888030020 con 0x7f98a00fc300 2024-09-10T19:01:30.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.729+0000 7f9897fff640 1 --2- 172.21.15.119:0/1696867784 >> v2:172.21.15.119:6800/1094129222 conn(0x7f988c06ede0 0x7f988c071290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:30.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.729+0000 7f989effd640 1 --2- 172.21.15.119:0/1696867784 >> v2:172.21.15.119:6800/1094129222 conn(0x7f988c06ede0 0x7f988c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:30.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.729+0000 7f9897fff640 1 -- 172.21.15.119:0/1696867784 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4077+0+0 (secure 0 0 0) 0x7f98880ad7a0 con 0x7f98a00fc300 2024-09-10T19:01:30.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.729+0000 7f989effd640 1 --2- 172.21.15.119:0/1696867784 >> v2:172.21.15.119:6800/1094129222 conn(0x7f988c06ede0 0x7f988c071290 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9890006fd0 tx=0x7f9890008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:30.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.733+0000 7f9897fff640 1 -- 172.21.15.119:0/1696867784 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f988807af70 con 0x7f98a00fc300 2024-09-10T19:01:30.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.889+0000 7f98a5343640 1 -- 172.21.15.119:0/1696867784 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98a010db20 con 0x7f98a00fc300 2024-09-10T19:01:30.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.890+0000 7f9897fff640 1 -- 172.21.15.119:0/1696867784 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+11181 (secure 0 0 0) 0x7f988807a910 con 0x7f98a00fc300 2024-09-10T19:01:30.892 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T19:01:30.892 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":50,"fsid":"3afe9804-6fa6-11ef-bcea-c7b262605968","created":"2024-09-10T18:56:26.299359+0000","modified":"2024-09-10T19:01:26.156198+0000","last_up_change":"2024-09-10T19:01:24.140998+0000","last_in_change":"2024-09-10T19:01:07.206749+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T18:59:30.453601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"7c2033fa-42d6-44b1-ac93-8e4ea2aea00f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6801","nonce":1598455131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1598455131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1598455131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6803","nonce":1598455131}]},"public_addr":"172.21.15.119:6801/1598455131","cluster_addr":"172.21.15.119:6802/1598455131","heartbeat_back_addr":"172.21.15.119:6804/1598455131","heartbeat_front_addr":"172.21.15.119:6803/1598455131","state":["exists","up"]},{"osd":1,"uuid":"cdc97929-345b-440d-9d0f-c64e3cbe8110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6805","nonce":4018071000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":4018071000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":4018071000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6807","nonce":4018071000}]},"public_addr":"172.21.15.119:6805/4018071000","cluster_addr":"172.21.15.119:6806/4018071000","heartbeat_back_addr":"172.21.15.119:6808/4018071000","heartbeat_front_addr":"172.21.15.119:6807/4018071000","state":["exists","up"]},{"osd":2,"uuid":"a53d3e5b-f03d-4082-a6d5-af183c1e8ca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6809","nonce":150641920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":150641920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":150641920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6811","nonce":150641920}]},"public_addr":"172.21.15.119:6809/150641920","cluster_addr":"172.21.15.119:6810/150641920","heartbeat_back_addr":"172.21.15.119:6812/150641920","heartbeat_front_addr":"172.21.15.119:6811/150641920","state":["exists","up"]},{"osd":3,"uuid":"291e81da-6fc3-44a4-880c-dd6c0ad00c13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6813","nonce":783850966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":783850966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":783850966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6815","nonce":783850966}]},"public_addr":"172.21.15.119:6813/783850966","cluster_addr":"172.21.15.119:6814/783850966","heartbeat_back_addr":"172.21.15.119:6816/783850966","heartbeat_front_addr":"172.21.15.119:6815/783850966","state":["exists","up"]},{"osd":4,"uuid":"0bee9361-6ab4-4545-a6d4-b05bbb4f9c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":4259870591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6801","nonce":4259870591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6803","nonce":4259870591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":4259870591}]},"public_addr":"172.21.15.194:6800/4259870591","cluster_addr":"172.21.15.194:6801/4259870591","heartbeat_back_addr":"172.21.15.194:6803/4259870591","heartbeat_front_addr":"172.21.15.194:6802/4259870591","state":["exists","up"]},{"osd":5,"uuid":"b26af90f-5d90-4929-94e6-92545dffd78d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":635822671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6805","nonce":635822671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6807","nonce":635822671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":635822671}]},"public_addr":"172.21.15.194:6804/635822671","cluster_addr":"172.21.15.194:6805/635822671","heartbeat_back_addr":"172.21.15.194:6807/635822671","heartbeat_front_addr":"172.21.15.194:6806/635822671","state":["exists","up"]},{"osd":6,"uuid":"c05bb66f-8bc0-44b3-965d-fda246401108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2830881352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6809","nonce":2830881352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6811","nonce":2830881352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2830881352}]},"public_addr":"172.21.15.194:6808/2830881352","cluster_addr":"172.21.15.194:6809/2830881352","heartbeat_back_addr":"172.21.15.194:6811/2830881352","heartbeat_front_addr":"172.21.15.194:6810/2830881352","state":["exists","up"]},{"osd":7,"uuid":"16049fee-bcde-450e-b4c5-de13dbb392d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":117762113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6813","nonce":117762113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6815","nonce":117762113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":117762113}]},"public_addr":"172.21.15.194:6812/117762113","cluster_addr":"172.21.15.194:6813/117762113","heartbeat_back_addr":"172.21.15.194:6815/117762113","heartbeat_front_addr":"172.21.15.194:6814/117762113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:58:39.672796+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:59:03.878834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:59:27.525213+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:59:51.755950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:00:12.576692+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:00:35.773984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:00:58.811225+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:01:22.187718+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/667233824":"2024-09-11T18:57:18.128849+0000","172.21.15.119:6800/1136125279":"2024-09-11T18:57:18.128849+0000","172.21.15.119:0/651881751":"2024-09-11T18:57:18.128849+0000","172.21.15.119:0/359148006":"2024-09-11T18:56:55.126452+0000","172.21.15.119:0/1889495967":"2024-09-11T18:57:18.128849+0000","172.21.15.119:0/3881701229":"2024-09-11T18:56:55.126452+0000","172.21.15.119:6800/4184327284":"2024-09-11T18:56:55.126452+0000","172.21.15.119:0/4285164139":"2024-09-11T18:56:55.126452+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:01:30.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.892+0000 7f98a5343640 1 -- 172.21.15.119:0/1696867784 >> v2:172.21.15.119:6800/1094129222 conn(0x7f988c06ede0 msgr2=0x7f988c071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:30.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.892+0000 7f98a5343640 1 --2- 172.21.15.119:0/1696867784 >> v2:172.21.15.119:6800/1094129222 conn(0x7f988c06ede0 0x7f988c071290 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9890006fd0 tx=0x7f9890008040 comp rx=0 tx=0).stop 2024-09-10T19:01:30.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.892+0000 7f98a5343640 1 -- 172.21.15.119:0/1696867784 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f98a00fc300 msgr2=0x7f98a0113390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:30.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.892+0000 7f98a5343640 1 --2- 172.21.15.119:0/1696867784 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f98a00fc300 0x7f98a0113390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f98880094b0 tx=0x7f9888009da0 comp rx=0 tx=0).stop 2024-09-10T19:01:30.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.892+0000 7f98a5343640 1 -- 172.21.15.119:0/1696867784 shutdown_connections 2024-09-10T19:01:30.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.892+0000 7f98a5343640 1 --2- 172.21.15.119:0/1696867784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98a01017b0 0x7f98a010cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:30.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.892+0000 7f98a5343640 1 --2- 172.21.15.119:0/1696867784 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f98a00fc300 0x7f98a0113390 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:30.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.892+0000 7f98a5343640 1 --2- 172.21.15.119:0/1696867784 >> v2:172.21.15.119:6800/1094129222 conn(0x7f988c06ede0 0x7f988c071290 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:30.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.892+0000 7f98a5343640 1 --2- 172.21.15.119:0/1696867784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98a00fb950 0x7f98a0112e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:30.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.892+0000 7f98a5343640 1 -- 172.21.15.119:0/1696867784 >> 172.21.15.119:0/1696867784 conn(0x7f98a00724d0 msgr2=0x7f98a0065f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:30.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.892+0000 7f98a5343640 1 -- 172.21.15.119:0/1696867784 shutdown_connections 2024-09-10T19:01:30.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:30.892+0000 7f98a5343640 1 -- 172.21.15.119:0/1696867784 wait complete. 2024-09-10T19:01:31.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:31 smithi119 ceph-mon[29577]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:31.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1696867784' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:01:31.513 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:31 smithi119 ceph-mon[40674]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:31.513 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1696867784' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:01:31.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:31 smithi194 ceph-mon[32464]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:31.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1696867784' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:01:31.537 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T18:59:30.453601+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-10T19:01:31.538 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-10T19:01:31.777 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:01:33.480 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:33 smithi119 ceph-mon[29577]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:33.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 -- 172.21.15.119:0/3602211211 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48901017d0 msgr2=0x7f4890101ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:33.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 --2- 172.21.15.119:0/3602211211 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48901017d0 0x7f4890101ba0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4880004980 tx=0x7f4880009da0 comp rx=0 tx=0).stop 2024-09-10T19:01:33.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 -- 172.21.15.119:0/3602211211 shutdown_connections 2024-09-10T19:01:33.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 --2- 172.21.15.119:0/3602211211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48900fc1b0 0x7f4890108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:33.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 --2- 172.21.15.119:0/3602211211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48900fb800 0x7f48900fbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:33.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 --2- 172.21.15.119:0/3602211211 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48901017d0 0x7f4890101ba0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:33.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 -- 172.21.15.119:0/3602211211 >> 172.21.15.119:0/3602211211 conn(0x7f4890071c00 msgr2=0x7f4890072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:33.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 -- 172.21.15.119:0/3602211211 shutdown_connections 2024-09-10T19:01:33.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 -- 172.21.15.119:0/3602211211 wait complete. 2024-09-10T19:01:33.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 Processor -- start 2024-09-10T19:01:33.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 -- start start 2024-09-10T19:01:33.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48900fb800 0x7f489010dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:33.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48900fc1b0 0x7f489010e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:33.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48901017d0 0x7f489010ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:33.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4890119210 con 0x7f48900fb800 2024-09-10T19:01:33.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4890119380 con 0x7f48900fc1b0 2024-09-10T19:01:33.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.477+0000 7f4895f93640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4890119540 con 0x7f48901017d0 2024-09-10T19:01:33.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f488f7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48900fb800 0x7f489010dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:33.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f488ffff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48901017d0 0x7f489010ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:33.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f488f7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48900fb800 0x7f489010dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51156/0 (socket says 172.21.15.119:51156) 2024-09-10T19:01:33.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f488f7fe640 1 -- 172.21.15.119:0/1286718479 learned_addr learned my addr 172.21.15.119:0/1286718479 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:01:33.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f4887fff640 1 --2- 172.21.15.119:0/1286718479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48900fc1b0 0x7f489010e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:33.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f488ffff640 1 -- 172.21.15.119:0/1286718479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48900fc1b0 msgr2=0x7f489010e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:33.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f488ffff640 1 --2- 172.21.15.119:0/1286718479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48900fc1b0 0x7f489010e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:33.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f488ffff640 1 -- 172.21.15.119:0/1286718479 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48900fb800 msgr2=0x7f489010dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:33.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f488ffff640 1 --2- 172.21.15.119:0/1286718479 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48900fb800 0x7f489010dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:33.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f488ffff640 1 -- 172.21.15.119:0/1286718479 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4880005040 con 0x7f48901017d0 2024-09-10T19:01:33.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f4887fff640 1 --2- 172.21.15.119:0/1286718479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48900fc1b0 0x7f489010e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T19:01:33.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f488f7fe640 1 --2- 172.21.15.119:0/1286718479 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48900fb800 0x7f489010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:01:33.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f488ffff640 1 --2- 172.21.15.119:0/1286718479 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48901017d0 0x7f489010ea60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f488800ebd0 tx=0x7f488800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:33.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f488cff9640 1 -- 172.21.15.119:0/1286718479 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4888004020 con 0x7f48901017d0 2024-09-10T19:01:33.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f4895f93640 1 -- 172.21.15.119:0/1286718479 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4890119820 con 0x7f48901017d0 2024-09-10T19:01:33.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f488cff9640 1 -- 172.21.15.119:0/1286718479 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f488800f040 con 0x7f48901017d0 2024-09-10T19:01:33.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f488cff9640 1 -- 172.21.15.119:0/1286718479 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488800ce00 con 0x7f48901017d0 2024-09-10T19:01:33.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.478+0000 7f4895f93640 1 -- 172.21.15.119:0/1286718479 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4890119d20 con 0x7f48901017d0 2024-09-10T19:01:33.484 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:33 smithi119 ceph-mon[40674]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:33.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.479+0000 7f4895f93640 1 -- 172.21.15.119:0/1286718479 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48900fcef0 con 0x7f48901017d0 2024-09-10T19:01:33.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.480+0000 7f488cff9640 1 -- 172.21.15.119:0/1286718479 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 15) v1 ==== 90746+0+0 (secure 0 0 0) 0x7f48880092e0 con 0x7f48901017d0 2024-09-10T19:01:33.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.480+0000 7f488cff9640 1 --2- 172.21.15.119:0/1286718479 >> v2:172.21.15.119:6800/1094129222 conn(0x7f487c06ede0 0x7f487c071290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:33.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.480+0000 7f488cff9640 1 -- 172.21.15.119:0/1286718479 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4077+0+0 (secure 0 0 0) 0x7f488808dfe0 con 0x7f48901017d0 2024-09-10T19:01:33.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.480+0000 7f488f7fe640 1 --2- 172.21.15.119:0/1286718479 >> v2:172.21.15.119:6800/1094129222 conn(0x7f487c06ede0 0x7f487c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:33.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.481+0000 7f488f7fe640 1 --2- 172.21.15.119:0/1286718479 >> v2:172.21.15.119:6800/1094129222 conn(0x7f487c06ede0 0x7f487c071290 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4880007ac0 tx=0x7f4880009da0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:33.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.485+0000 7f488cff9640 1 -- 172.21.15.119:0/1286718479 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f488805b750 con 0x7f48901017d0 2024-09-10T19:01:33.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:33 smithi194 ceph-mon[32464]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:33.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.640+0000 7f4895f93640 1 -- 172.21.15.119:0/1286718479 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f489010acd0 con 0x7f48901017d0 2024-09-10T19:01:33.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.640+0000 7f488cff9640 1 -- 172.21.15.119:0/1286718479 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f48880180a0 con 0x7f48901017d0 2024-09-10T19:01:33.642 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-10T19:01:33.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.642+0000 7f4895f93640 1 -- 172.21.15.119:0/1286718479 >> v2:172.21.15.119:6800/1094129222 conn(0x7f487c06ede0 msgr2=0x7f487c071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:33.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.642+0000 7f4895f93640 1 --2- 172.21.15.119:0/1286718479 >> v2:172.21.15.119:6800/1094129222 conn(0x7f487c06ede0 0x7f487c071290 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4880007ac0 tx=0x7f4880009da0 comp rx=0 tx=0).stop 2024-09-10T19:01:33.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.642+0000 7f4895f93640 1 -- 172.21.15.119:0/1286718479 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48901017d0 msgr2=0x7f489010ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:33.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.642+0000 7f4895f93640 1 --2- 172.21.15.119:0/1286718479 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48901017d0 0x7f489010ea60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f488800ebd0 tx=0x7f488800b040 comp rx=0 tx=0).stop 2024-09-10T19:01:33.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.643+0000 7f4895f93640 1 -- 172.21.15.119:0/1286718479 shutdown_connections 2024-09-10T19:01:33.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.643+0000 7f4895f93640 1 --2- 172.21.15.119:0/1286718479 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f48901017d0 0x7f489010ea60 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:33.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.643+0000 7f4895f93640 1 --2- 172.21.15.119:0/1286718479 >> v2:172.21.15.119:6800/1094129222 conn(0x7f487c06ede0 0x7f487c071290 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:33.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.643+0000 7f4895f93640 1 --2- 172.21.15.119:0/1286718479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48900fc1b0 0x7f489010e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:33.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.643+0000 7f4895f93640 1 --2- 172.21.15.119:0/1286718479 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48900fb800 0x7f489010dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:33.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.643+0000 7f4895f93640 1 -- 172.21.15.119:0/1286718479 >> 172.21.15.119:0/1286718479 conn(0x7f4890071c00 msgr2=0x7f48900fa180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:33.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.643+0000 7f4895f93640 1 -- 172.21.15.119:0/1286718479 shutdown_connections 2024-09-10T19:01:33.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:01:33.643+0000 7f4895f93640 1 -- 172.21.15.119:0/1286718479 wait complete. 2024-09-10T19:01:34.285 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi119 2024-09-10T19:01:34.285 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi119=foo.a' 2024-09-10T19:01:34.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1286718479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-10T19:01:34.540 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:01:34.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1286718479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-10T19:01:34.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1286718479' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-10T19:01:35.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:35 smithi194 ceph-mon[32464]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:35.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:35 smithi119 ceph-mon[29577]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:35.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:35 smithi119 ceph-mon[40674]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:36.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.109+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3244510642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ea81024c0 msgr2=0x7f7ea8104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:36.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.109+0000 7f7eae2e3640 1 --2- 172.21.15.194:0/3244510642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ea81024c0 0x7f7ea8104970 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0009cd0 tx=0x7f7ea0004d10 comp rx=0 tx=0).stop 2024-09-10T19:01:36.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.109+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3244510642 shutdown_connections 2024-09-10T19:01:36.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.109+0000 7f7eae2e3640 1 --2- 172.21.15.194:0/3244510642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ea81024c0 0x7f7ea8104970 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:36.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.109+0000 7f7eae2e3640 1 --2- 172.21.15.194:0/3244510642 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ea80ff9b0 0x7f7ea8101d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:36.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.109+0000 7f7eae2e3640 1 --2- 172.21.15.194:0/3244510642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ea8065f10 0x7f7ea80ff470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:36.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.109+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3244510642 >> 172.21.15.194:0/3244510642 conn(0x7f7ea80f6f20 msgr2=0x7f7ea80f9350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:36.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.109+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3244510642 shutdown_connections 2024-09-10T19:01:36.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.109+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3244510642 wait complete. 2024-09-10T19:01:36.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.109+0000 7f7eae2e3640 1 Processor -- start 2024-09-10T19:01:36.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.109+0000 7f7eae2e3640 1 -- start start 2024-09-10T19:01:36.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eae2e3640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ea8065f10 0x7f7ea810cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:36.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eae2e3640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ea80ff9b0 0x7f7ea810d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:36.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eae2e3640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ea81024c0 0x7f7ea8114060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:36.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eae2e3640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea8114630 con 0x7f7ea80ff9b0 2024-09-10T19:01:36.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eae2e3640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea81147a0 con 0x7f7ea81024c0 2024-09-10T19:01:36.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eae2e3640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ea8114960 con 0x7f7ea8065f10 2024-09-10T19:01:36.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eac880640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ea81024c0 0x7f7ea8114060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:36.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eac880640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ea81024c0 0x7f7ea8114060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:49528/0 (socket says 172.21.15.194:49528) 2024-09-10T19:01:36.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eac880640 1 -- 172.21.15.194:0/3606793873 learned_addr learned my addr 172.21.15.194:0/3606793873 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T19:01:36.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7ea77fe640 1 --2- 172.21.15.194:0/3606793873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ea80ff9b0 0x7f7ea810d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:36.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7ea7fff640 1 --2- 172.21.15.194:0/3606793873 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ea8065f10 0x7f7ea810cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:36.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eac880640 1 -- 172.21.15.194:0/3606793873 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ea8065f10 msgr2=0x7f7ea810cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:36.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eac880640 1 --2- 172.21.15.194:0/3606793873 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ea8065f10 0x7f7ea810cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:36.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eac880640 1 -- 172.21.15.194:0/3606793873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ea80ff9b0 msgr2=0x7f7ea810d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:36.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eac880640 1 --2- 172.21.15.194:0/3606793873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ea80ff9b0 0x7f7ea810d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:36.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eac880640 1 -- 172.21.15.194:0/3606793873 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea00098b0 con 0x7f7ea81024c0 2024-09-10T19:01:36.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eac880640 1 --2- 172.21.15.194:0/3606793873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ea81024c0 0x7f7ea8114060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0004750 tx=0x7f7ea0004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:36.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7ea4ff9640 1 -- 172.21.15.194:0/3606793873 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea0038070 con 0x7f7ea81024c0 2024-09-10T19:01:36.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7ea4ff9640 1 -- 172.21.15.194:0/3606793873 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f7ea00096e0 con 0x7f7ea81024c0 2024-09-10T19:01:36.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3606793873 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ea8114be0 con 0x7f7ea81024c0 2024-09-10T19:01:36.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.110+0000 7f7ea4ff9640 1 -- 172.21.15.194:0/3606793873 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea003cca0 con 0x7f7ea81024c0 2024-09-10T19:01:36.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.111+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3606793873 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ea81150b0 con 0x7f7ea81024c0 2024-09-10T19:01:36.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.112+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3606793873 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ea804bf60 con 0x7f7ea81024c0 2024-09-10T19:01:36.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.112+0000 7f7ea4ff9640 1 -- 172.21.15.194:0/3606793873 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 15) v1 ==== 90746+0+0 (secure 0 0 0) 0x7f7ea0030070 con 0x7f7ea81024c0 2024-09-10T19:01:36.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.112+0000 7f7ea4ff9640 1 --2- 172.21.15.194:0/3606793873 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7e9006ee10 0x7f7e900712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:36.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.112+0000 7f7ea4ff9640 1 -- 172.21.15.194:0/3606793873 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4077+0+0 (secure 0 0 0) 0x7f7ea00ae320 con 0x7f7ea81024c0 2024-09-10T19:01:36.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.113+0000 7f7ea7fff640 1 --2- 172.21.15.194:0/3606793873 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7e9006ee10 0x7f7e900712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:36.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.113+0000 7f7ea7fff640 1 --2- 172.21.15.194:0/3606793873 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7e9006ee10 0x7f7e900712c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7e98009b80 tx=0x7f7e9800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:36.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.117+0000 7f7ea4ff9640 1 -- 172.21.15.194:0/3606793873 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f7ea007ba60 con 0x7f7ea81024c0 2024-09-10T19:01:36.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.281+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3606793873 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi119=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ea8107830 con 0x7f7e9006ee10 2024-09-10T19:01:36.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.289+0000 7f7ea4ff9640 1 -- 172.21.15.194:0/3606793873 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (secure 0 0 0) 0x7f7ea8107830 con 0x7f7e9006ee10 2024-09-10T19:01:36.290 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled rgw.foo.a update... 2024-09-10T19:01:36.292 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.291+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3606793873 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7e9006ee10 msgr2=0x7f7e900712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:36.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.291+0000 7f7eae2e3640 1 --2- 172.21.15.194:0/3606793873 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7e9006ee10 0x7f7e900712c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7e98009b80 tx=0x7f7e9800b040 comp rx=0 tx=0).stop 2024-09-10T19:01:36.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.291+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3606793873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ea81024c0 msgr2=0x7f7ea8114060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:36.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.291+0000 7f7eae2e3640 1 --2- 172.21.15.194:0/3606793873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ea81024c0 0x7f7ea8114060 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7ea0004750 tx=0x7f7ea0004d10 comp rx=0 tx=0).stop 2024-09-10T19:01:36.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.291+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3606793873 shutdown_connections 2024-09-10T19:01:36.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.291+0000 7f7eae2e3640 1 --2- 172.21.15.194:0/3606793873 >> v2:172.21.15.119:6800/1094129222 conn(0x7f7e9006ee10 0x7f7e900712c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:36.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.291+0000 7f7eae2e3640 1 --2- 172.21.15.194:0/3606793873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ea81024c0 0x7f7ea8114060 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:36.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.291+0000 7f7eae2e3640 1 --2- 172.21.15.194:0/3606793873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ea80ff9b0 0x7f7ea810d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:36.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.291+0000 7f7eae2e3640 1 --2- 172.21.15.194:0/3606793873 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7ea8065f10 0x7f7ea810cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:36.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.291+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3606793873 >> 172.21.15.194:0/3606793873 conn(0x7f7ea80f6f20 msgr2=0x7f7ea80fdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:36.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.291+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3606793873 shutdown_connections 2024-09-10T19:01:36.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:36.291+0000 7f7eae2e3640 1 -- 172.21.15.194:0/3606793873 wait complete. 2024-09-10T19:01:36.879 DEBUG:teuthology.orchestra.run.smithi119:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@rgw.foo.a.service 2024-09-10T19:01:36.881 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi194 2024-09-10T19:01:36.882 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-10T19:01:37.123 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:01:37.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:37 smithi194 ceph-mon[32464]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:37.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:37 smithi194 ceph-mon[32464]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi119=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:37.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:37 smithi194 ceph-mon[32464]: Saving service rgw.foo.a spec with placement smithi119=foo.a;count:1 2024-09-10T19:01:37.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:37 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:37.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:37 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:37.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:37 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:37.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:37 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:37.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:37 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:37.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:37 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-10T19:01:37.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:37 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-10T19:01:37.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:37 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:37.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:37 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:37.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:37 smithi194 ceph-mon[32464]: Deploying daemon rgw.foo.a on smithi119 2024-09-10T19:01:37.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[29577]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:37.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[29577]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi119=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:37.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[29577]: Saving service rgw.foo.a spec with placement smithi119=foo.a;count:1 2024-09-10T19:01:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-10T19:01:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-10T19:01:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[29577]: Deploying daemon rgw.foo.a on smithi119 2024-09-10T19:01:37.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[40674]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:37.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[40674]: from='client.24328 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi119=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:37.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[40674]: Saving service rgw.foo.a spec with placement smithi119=foo.a;count:1 2024-09-10T19:01:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-10T19:01:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-10T19:01:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:37 smithi119 ceph-mon[40674]: Deploying daemon rgw.foo.a on smithi119 2024-09-10T19:01:38.566 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:01:38 smithi119 podman[68752]: 2024-09-10 19:01:38.475698404 +0000 UTC m=+0.139766645 container create a3ed74e0e69880090c038472ffc6319548a1b31baa7921e627b36553ae9c9f91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-rgw-foo-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:01:38.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.667+0000 7f880716d640 1 -- 172.21.15.194:0/832535199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88000fe4d0 msgr2=0x7f8800104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:38.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.667+0000 7f880716d640 1 --2- 172.21.15.194:0/832535199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88000fe4d0 0x7f8800104d70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f87f8009cd0 tx=0x7f87f8004d10 comp rx=0 tx=0).stop 2024-09-10T19:01:38.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.667+0000 7f880716d640 1 -- 172.21.15.194:0/832535199 shutdown_connections 2024-09-10T19:01:38.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.667+0000 7f880716d640 1 --2- 172.21.15.194:0/832535199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88000fe4d0 0x7f8800104d70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:38.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.667+0000 7f880716d640 1 --2- 172.21.15.194:0/832535199 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88000fdb20 0x7f88000fdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:38.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.667+0000 7f880716d640 1 --2- 172.21.15.194:0/832535199 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88000fc850 0x7f88000fcc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:38.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.667+0000 7f880716d640 1 -- 172.21.15.194:0/832535199 >> 172.21.15.194:0/832535199 conn(0x7f88000f8010 msgr2=0x7f88000fa440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:38.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.667+0000 7f880716d640 1 -- 172.21.15.194:0/832535199 shutdown_connections 2024-09-10T19:01:38.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.667+0000 7f880716d640 1 -- 172.21.15.194:0/832535199 wait complete. 2024-09-10T19:01:38.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.668+0000 7f880716d640 1 Processor -- start 2024-09-10T19:01:38.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.668+0000 7f880716d640 1 -- start start 2024-09-10T19:01:38.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.668+0000 7f880716d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88000fc850 0x7f880010cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:38.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.668+0000 7f880716d640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88000fdb20 0x7f880010d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:38.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.668+0000 7f880716d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88000fe4d0 0x7f8800114150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:38.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.668+0000 7f880716d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8800114720 con 0x7f88000fc850 2024-09-10T19:01:38.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.668+0000 7f880716d640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8800114890 con 0x7f88000fdb20 2024-09-10T19:01:38.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.668+0000 7f880716d640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8800114a50 con 0x7f88000fe4d0 2024-09-10T19:01:38.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.668+0000 7f87fffff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88000fdb20 0x7f880010d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:38.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.668+0000 7f87fffff640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88000fdb20 0x7f880010d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:49554/0 (socket says 172.21.15.194:49554) 2024-09-10T19:01:38.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.668+0000 7f87fffff640 1 -- 172.21.15.194:0/4176901381 learned_addr learned my addr 172.21.15.194:0/4176901381 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T19:01:38.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.668+0000 7f8804f09640 1 --2- 172.21.15.194:0/4176901381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88000fc850 0x7f880010cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:38.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.668+0000 7f880570a640 1 --2- 172.21.15.194:0/4176901381 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88000fe4d0 0x7f8800114150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:38.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.669+0000 7f87fffff640 1 -- 172.21.15.194:0/4176901381 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88000fe4d0 msgr2=0x7f8800114150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:38.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.669+0000 7f87fffff640 1 --2- 172.21.15.194:0/4176901381 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88000fe4d0 0x7f8800114150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:38.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.669+0000 7f87fffff640 1 -- 172.21.15.194:0/4176901381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88000fc850 msgr2=0x7f880010cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:38.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.669+0000 7f87fffff640 1 --2- 172.21.15.194:0/4176901381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88000fc850 0x7f880010cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:38.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.669+0000 7f87fffff640 1 -- 172.21.15.194:0/4176901381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f80098b0 con 0x7f88000fdb20 2024-09-10T19:01:38.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.669+0000 7f87fffff640 1 --2- 172.21.15.194:0/4176901381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88000fdb20 0x7f880010d0f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00b810 tx=0x7f87ec0091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:38.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.669+0000 7f87fd7fa640 1 -- 172.21.15.194:0/4176901381 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec00be60 con 0x7f88000fdb20 2024-09-10T19:01:38.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.669+0000 7f87fd7fa640 1 -- 172.21.15.194:0/4176901381 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f87ec00cba0 con 0x7f88000fdb20 2024-09-10T19:01:38.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.669+0000 7f87fd7fa640 1 -- 172.21.15.194:0/4176901381 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec015b60 con 0x7f88000fdb20 2024-09-10T19:01:38.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.669+0000 7f880716d640 1 -- 172.21.15.194:0/4176901381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8800114d30 con 0x7f88000fdb20 2024-09-10T19:01:38.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.669+0000 7f880716d640 1 -- 172.21.15.194:0/4176901381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88001151a0 con 0x7f88000fdb20 2024-09-10T19:01:38.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.670+0000 7f880716d640 1 -- 172.21.15.194:0/4176901381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f880004bf60 con 0x7f88000fdb20 2024-09-10T19:01:38.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.670+0000 7f87fd7fa640 1 -- 172.21.15.194:0/4176901381 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 15) v1 ==== 90746+0+0 (secure 0 0 0) 0x7f87ec012070 con 0x7f88000fdb20 2024-09-10T19:01:38.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.671+0000 7f87fd7fa640 1 --2- 172.21.15.194:0/4176901381 >> v2:172.21.15.119:6800/1094129222 conn(0x7f87f006ee10 0x7f87f00712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:38.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.671+0000 7f87fd7fa640 1 -- 172.21.15.194:0/4176901381 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 4077+0+0 (secure 0 0 0) 0x7f87ec08d160 con 0x7f88000fdb20 2024-09-10T19:01:38.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.671+0000 7f8804f09640 1 --2- 172.21.15.194:0/4176901381 >> v2:172.21.15.119:6800/1094129222 conn(0x7f87f006ee10 0x7f87f00712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:38.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.671+0000 7f8804f09640 1 --2- 172.21.15.194:0/4176901381 >> v2:172.21.15.119:6800/1094129222 conn(0x7f87f006ee10 0x7f87f00712c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f87f40041d0 tx=0x7f87f4004330 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:38.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.675+0000 7f87fd7fa640 1 -- 172.21.15.194:0/4176901381 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f87ec05a920 con 0x7f88000fdb20 2024-09-10T19:01:38.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:38.832+0000 7f880716d640 1 -- 172.21.15.194:0/4176901381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7f880002aa40 con 0x7f88000fdb20 2024-09-10T19:01:39.066 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:01:38 smithi119 podman[68752]: 2024-09-10 19:01:38.728053349 +0000 UTC m=+0.392121591 container init a3ed74e0e69880090c038472ffc6319548a1b31baa7921e627b36553ae9c9f91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-10T19:01:39.067 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:01:38 smithi119 podman[68752]: 2024-09-10 19:01:38.732882206 +0000 UTC m=+0.396950449 container start a3ed74e0e69880090c038472ffc6319548a1b31baa7921e627b36553ae9c9f91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-rgw-foo-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731) 2024-09-10T19:01:39.067 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:01:38 smithi119 bash[68752]: a3ed74e0e69880090c038472ffc6319548a1b31baa7921e627b36553ae9c9f91 2024-09-10T19:01:39.067 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:01:38 smithi119 systemd[1]: Started Ceph rgw.foo.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:01:39.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:39.299+0000 7f87fd7fa640 1 -- 172.21.15.194:0/4176901381 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v51) v1 ==== 160+0+0 (secure 0 0 0) 0x7f87ec05a2c0 con 0x7f88000fdb20 2024-09-10T19:01:39.301 INFO:teuthology.orchestra.run.smithi194.stderr:pool 'datapool' created 2024-09-10T19:01:39.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:39.303+0000 7f880716d640 1 -- 172.21.15.194:0/4176901381 >> v2:172.21.15.119:6800/1094129222 conn(0x7f87f006ee10 msgr2=0x7f87f00712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:39.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:39.303+0000 7f880716d640 1 --2- 172.21.15.194:0/4176901381 >> v2:172.21.15.119:6800/1094129222 conn(0x7f87f006ee10 0x7f87f00712c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f87f40041d0 tx=0x7f87f4004330 comp rx=0 tx=0).stop 2024-09-10T19:01:39.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:39.303+0000 7f880716d640 1 -- 172.21.15.194:0/4176901381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88000fdb20 msgr2=0x7f880010d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:39.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:39.303+0000 7f880716d640 1 --2- 172.21.15.194:0/4176901381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88000fdb20 0x7f880010d0f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f87ec00b810 tx=0x7f87ec0091d0 comp rx=0 tx=0).stop 2024-09-10T19:01:39.305 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:39.303+0000 7f880716d640 1 -- 172.21.15.194:0/4176901381 shutdown_connections 2024-09-10T19:01:39.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:39.304+0000 7f880716d640 1 --2- 172.21.15.194:0/4176901381 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f88000fe4d0 0x7f8800114150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:39.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:39.304+0000 7f880716d640 1 --2- 172.21.15.194:0/4176901381 >> v2:172.21.15.119:6800/1094129222 conn(0x7f87f006ee10 0x7f87f00712c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:39.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:39.304+0000 7f880716d640 1 --2- 172.21.15.194:0/4176901381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88000fdb20 0x7f880010d0f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:39.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:39.304+0000 7f880716d640 1 --2- 172.21.15.194:0/4176901381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88000fc850 0x7f880010cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:39.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:39.304+0000 7f880716d640 1 -- 172.21.15.194:0/4176901381 >> 172.21.15.194:0/4176901381 conn(0x7f88000f8010 msgr2=0x7f88000f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:39.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:39.304+0000 7f880716d640 1 -- 172.21.15.194:0/4176901381 shutdown_connections 2024-09-10T19:01:39.306 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:39.304+0000 7f880716d640 1 -- 172.21.15.194:0/4176901381 wait complete. 2024-09-10T19:01:39.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:39 smithi194 ceph-mon[32464]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:39.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:39 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/4176901381' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T19:01:39.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T19:01:39.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:39 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:39 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:39 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:39 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:39 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[29577]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:39.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/4176901381' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T19:01:39.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T19:01:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[40674]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/4176901381' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:39 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:39.846 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- rbd pool init datapool 2024-09-10T19:01:40.085 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:01:40.303 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:40 smithi119 ceph-mon[29577]: Saving service rgw.foo.a spec with placement smithi119=foo.a;count:1 2024-09-10T19:01:40.459 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:40 smithi194 ceph-mon[32464]: Saving service rgw.foo.a spec with placement smithi119=foo.a;count:1 2024-09-10T19:01:40.459 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:40 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-10T19:01:40.459 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:40 smithi194 ceph-mon[32464]: osdmap e51: 8 total, 8 up, 8 in 2024-09-10T19:01:40.459 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T19:01:40.459 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:40 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:40.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:40 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-10T19:01:40.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:40 smithi119 ceph-mon[29577]: osdmap e51: 8 total, 8 up, 8 in 2024-09-10T19:01:40.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T19:01:40.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:40 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:40.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:40 smithi119 ceph-mon[40674]: Saving service rgw.foo.a spec with placement smithi119=foo.a;count:1 2024-09-10T19:01:40.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:40 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-10T19:01:40.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:40 smithi119 ceph-mon[40674]: osdmap e51: 8 total, 8 up, 8 in 2024-09-10T19:01:40.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T19:01:40.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:40 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:41.472 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:41 smithi194 ceph-mon[32464]: pgmap v172: 36 pgs: 4 creating+peering, 31 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:41.473 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:41 smithi194 ceph-mon[32464]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:41.473 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-10T19:01:41.473 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:41 smithi194 ceph-mon[32464]: osdmap e52: 8 total, 8 up, 8 in 2024-09-10T19:01:41.473 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:41 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:41.473 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:41 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:41.473 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:41 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:41.473 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:41 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:41.473 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:41 smithi194 ceph-mon[32464]: Checking dashboard <-> RGW credentials 2024-09-10T19:01:41.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[29577]: pgmap v172: 36 pgs: 4 creating+peering, 31 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:41.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[29577]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:41.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-10T19:01:41.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[29577]: osdmap e52: 8 total, 8 up, 8 in 2024-09-10T19:01:41.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:41.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:41.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:41.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:41.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[29577]: Checking dashboard <-> RGW credentials 2024-09-10T19:01:41.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[40674]: pgmap v172: 36 pgs: 4 creating+peering, 31 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:01:41.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[40674]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:41.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-10T19:01:41.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[40674]: osdmap e52: 8 total, 8 up, 8 in 2024-09-10T19:01:41.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:41.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:41.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:41.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:41.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:41 smithi119 ceph-mon[40674]: Checking dashboard <-> RGW credentials 2024-09-10T19:01:42.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:42 smithi119 ceph-mon[29577]: osdmap e53: 8 total, 8 up, 8 in 2024-09-10T19:01:42.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3499169372' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:01:42.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:01:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/574265530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T19:01:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T19:01:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:42 smithi119 ceph-mon[40674]: osdmap e53: 8 total, 8 up, 8 in 2024-09-10T19:01:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3499169372' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:01:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:01:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/574265530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T19:01:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T19:01:42.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:42 smithi194 ceph-mon[32464]: osdmap e53: 8 total, 8 up, 8 in 2024-09-10T19:01:42.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3499169372' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:01:42.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:01:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/574265530' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T19:01:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T19:01:43.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:43 smithi194 ceph-mon[32464]: pgmap v175: 68 pgs: 12 creating+peering, 39 unknown, 17 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 170 B/s wr, 0 op/s 2024-09-10T19:01:43.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3499169372' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T19:01:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T19:01:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-10T19:01:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:43 smithi194 ceph-mon[32464]: osdmap e54: 8 total, 8 up, 8 in 2024-09-10T19:01:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:43 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:43.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:43 smithi119 ceph-mon[29577]: pgmap v175: 68 pgs: 12 creating+peering, 39 unknown, 17 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 170 B/s wr, 0 op/s 2024-09-10T19:01:43.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3499169372' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T19:01:43.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T19:01:43.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-10T19:01:43.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:43 smithi119 ceph-mon[29577]: osdmap e54: 8 total, 8 up, 8 in 2024-09-10T19:01:43.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:43 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:43.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:43 smithi119 ceph-mon[40674]: pgmap v175: 68 pgs: 12 creating+peering, 39 unknown, 17 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 511 B/s rd, 170 B/s wr, 0 op/s 2024-09-10T19:01:43.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3499169372' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T19:01:43.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T19:01:43.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-10T19:01:43.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:43 smithi119 ceph-mon[40674]: osdmap e54: 8 total, 8 up, 8 in 2024-09-10T19:01:43.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:43 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:44.641 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:44 smithi194 ceph-mon[32464]: osdmap e55: 8 total, 8 up, 8 in 2024-09-10T19:01:44.641 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3499169372' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:01:44.641 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:01:44.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:44 smithi119 ceph-mon[29577]: osdmap e55: 8 total, 8 up, 8 in 2024-09-10T19:01:44.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3499169372' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:01:44.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:01:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:44 smithi119 ceph-mon[40674]: osdmap e55: 8 total, 8 up, 8 in 2024-09-10T19:01:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3499169372' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:01:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:01:44.937 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.194 --placement '1;smithi194=iscsi.a' 2024-09-10T19:01:45.186 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:01:45.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:45 smithi194 ceph-mon[32464]: pgmap v178: 100 pgs: 20 creating+peering, 32 unknown, 48 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-10T19:01:45.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3499169372' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T19:01:45.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T19:01:45.586 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:45 smithi194 ceph-mon[32464]: osdmap e56: 8 total, 8 up, 8 in 2024-09-10T19:01:45.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:45 smithi119 ceph-mon[29577]: pgmap v178: 100 pgs: 20 creating+peering, 32 unknown, 48 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-10T19:01:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3499169372' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T19:01:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T19:01:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:45 smithi119 ceph-mon[29577]: osdmap e56: 8 total, 8 up, 8 in 2024-09-10T19:01:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:45 smithi119 ceph-mon[40674]: pgmap v178: 100 pgs: 20 creating+peering, 32 unknown, 48 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-10T19:01:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3499169372' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T19:01:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1751813100' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T19:01:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:45 smithi119 ceph-mon[40674]: osdmap e56: 8 total, 8 up, 8 in 2024-09-10T19:01:46.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:46 smithi194 ceph-mon[32464]: osdmap e57: 8 total, 8 up, 8 in 2024-09-10T19:01:46.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2161474448' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:01:46.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1419775440' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:01:46.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:46 smithi194 ceph-mon[32464]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:01:46.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:46 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:46.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1419775440' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T19:01:46.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:46 smithi194 ceph-mon[32464]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T19:01:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:46 smithi194 ceph-mon[32464]: osdmap e58: 8 total, 8 up, 8 in 2024-09-10T19:01:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1419775440' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:01:46.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.734+0000 7f6f62f26640 1 -- 172.21.15.194:0/3778369215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f5c0fdaf0 msgr2=0x7f6f5c0fdf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.734+0000 7f6f62f26640 1 --2- 172.21.15.194:0/3778369215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f5c0fdaf0 0x7f6f5c0fdf60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6f48004950 tx=0x7f6f48009da0 comp rx=0 tx=0).stop 2024-09-10T19:01:46.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.734+0000 7f6f62f26640 1 -- 172.21.15.194:0/3778369215 shutdown_connections 2024-09-10T19:01:46.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.734+0000 7f6f62f26640 1 --2- 172.21.15.194:0/3778369215 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f5c0fe4a0 0x7f6f5c104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:46.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.734+0000 7f6f62f26640 1 --2- 172.21.15.194:0/3778369215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f5c0fdaf0 0x7f6f5c0fdf60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:46.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.734+0000 7f6f62f26640 1 --2- 172.21.15.194:0/3778369215 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6f5c0fc820 0x7f6f5c0fcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:46.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.734+0000 7f6f62f26640 1 -- 172.21.15.194:0/3778369215 >> 172.21.15.194:0/3778369215 conn(0x7f6f5c0f7fc0 msgr2=0x7f6f5c0fa410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:46.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.734+0000 7f6f62f26640 1 -- 172.21.15.194:0/3778369215 shutdown_connections 2024-09-10T19:01:46.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.734+0000 7f6f62f26640 1 -- 172.21.15.194:0/3778369215 wait complete. 2024-09-10T19:01:46.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f62f26640 1 Processor -- start 2024-09-10T19:01:46.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f62f26640 1 -- start start 2024-09-10T19:01:46.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f62f26640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f5c0fc820 0x7f6f5c0724a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:46.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f62f26640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f5c0fdaf0 0x7f6f5c070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:46.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f62f26640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6f5c0fe4a0 0x7f6f5c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:46.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f62f26640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c0717f0 con 0x7f6f5c0fdaf0 2024-09-10T19:01:46.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f62f26640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c071960 con 0x7f6f5c0fc820 2024-09-10T19:01:46.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f62f26640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f5c116e10 con 0x7f6f5c0fe4a0 2024-09-10T19:01:46.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f60cc2640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f5c0fc820 0x7f6f5c0724a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:46.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f60cc2640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f5c0fc820 0x7f6f5c0724a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:34838/0 (socket says 172.21.15.194:34838) 2024-09-10T19:01:46.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f60cc2640 1 -- 172.21.15.194:0/3731875478 learned_addr learned my addr 172.21.15.194:0/3731875478 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T19:01:46.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f5bfff640 1 --2- 172.21.15.194:0/3731875478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f5c0fdaf0 0x7f6f5c070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:46.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f614c3640 1 --2- 172.21.15.194:0/3731875478 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6f5c0fe4a0 0x7f6f5c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:46.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f60cc2640 1 -- 172.21.15.194:0/3731875478 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6f5c0fe4a0 msgr2=0x7f6f5c0710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f60cc2640 1 --2- 172.21.15.194:0/3731875478 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6f5c0fe4a0 0x7f6f5c0710b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:46.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f60cc2640 1 -- 172.21.15.194:0/3731875478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f5c0fdaf0 msgr2=0x7f6f5c070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f60cc2640 1 --2- 172.21.15.194:0/3731875478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f5c0fdaf0 0x7f6f5c070b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:46.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f60cc2640 1 -- 172.21.15.194:0/3731875478 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f48005040 con 0x7f6f5c0fc820 2024-09-10T19:01:46.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.735+0000 7f6f614c3640 1 --2- 172.21.15.194:0/3731875478 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6f5c0fe4a0 0x7f6f5c0710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T19:01:46.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.736+0000 7f6f60cc2640 1 --2- 172.21.15.194:0/3731875478 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f5c0fc820 0x7f6f5c0724a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6f5000c810 tx=0x7f6f500043c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:46.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.736+0000 7f6f597fa640 1 -- 172.21.15.194:0/3731875478 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5000ce00 con 0x7f6f5c0fc820 2024-09-10T19:01:46.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.736+0000 7f6f597fa640 1 -- 172.21.15.194:0/3731875478 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f6f5000ac60 con 0x7f6f5c0fc820 2024-09-10T19:01:46.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.736+0000 7f6f62f26640 1 -- 172.21.15.194:0/3731875478 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f5c117070 con 0x7f6f5c0fc820 2024-09-10T19:01:46.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.736+0000 7f6f597fa640 1 -- 172.21.15.194:0/3731875478 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f50012bc0 con 0x7f6f5c0fc820 2024-09-10T19:01:46.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.736+0000 7f6f62f26640 1 -- 172.21.15.194:0/3731875478 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5c117540 con 0x7f6f5c0fc820 2024-09-10T19:01:46.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.737+0000 7f6f62f26640 1 -- 172.21.15.194:0/3731875478 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5c04bf60 con 0x7f6f5c0fc820 2024-09-10T19:01:46.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.737+0000 7f6f597fa640 1 -- 172.21.15.194:0/3731875478 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 15) v1 ==== 90746+0+0 (secure 0 0 0) 0x7f6f5000a7f0 con 0x7f6f5c0fc820 2024-09-10T19:01:46.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.738+0000 7f6f597fa640 1 --2- 172.21.15.194:0/3731875478 >> v2:172.21.15.119:6800/1094129222 conn(0x7f6f4c06ede0 0x7f6f4c071290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:46.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.738+0000 7f6f597fa640 1 -- 172.21.15.194:0/3731875478 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5922+0+0 (secure 0 0 0) 0x7f6f5008e1c0 con 0x7f6f5c0fc820 2024-09-10T19:01:46.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.738+0000 7f6f5bfff640 1 --2- 172.21.15.194:0/3731875478 >> v2:172.21.15.119:6800/1094129222 conn(0x7f6f4c06ede0 0x7f6f4c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:46.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.738+0000 7f6f5bfff640 1 --2- 172.21.15.194:0/3731875478 >> v2:172.21.15.119:6800/1094129222 conn(0x7f6f4c06ede0 0x7f6f4c071290 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6f480094b0 tx=0x7f6f48035040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:46.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.742+0000 7f6f597fa640 1 -- 172.21.15.194:0/3731875478 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f6f5005b190 con 0x7f6f5c0fc820 2024-09-10T19:01:46.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[29577]: osdmap e57: 8 total, 8 up, 8 in 2024-09-10T19:01:46.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2161474448' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:01:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1419775440' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:01:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[29577]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:01:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1419775440' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T19:01:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[29577]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T19:01:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[29577]: osdmap e58: 8 total, 8 up, 8 in 2024-09-10T19:01:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1419775440' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:01:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[40674]: osdmap e57: 8 total, 8 up, 8 in 2024-09-10T19:01:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2161474448' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:01:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1419775440' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:01:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[40674]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:01:46.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:01:46.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1419775440' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T19:01:46.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[40674]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T19:01:46.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[40674]: osdmap e58: 8 total, 8 up, 8 in 2024-09-10T19:01:46.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1419775440' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:01:46.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.906+0000 7f6f62f26640 1 -- 172.21.15.194:0/3731875478 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.194", "placement": "1;smithi194=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f5c05db50 con 0x7f6f4c06ede0 2024-09-10T19:01:46.917 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled iscsi.datapool update... 2024-09-10T19:01:46.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.915+0000 7f6f597fa640 1 -- 172.21.15.194:0/3731875478 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (secure 0 0 0) 0x7f6f5c05db50 con 0x7f6f4c06ede0 2024-09-10T19:01:46.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.917+0000 7f6f62f26640 1 -- 172.21.15.194:0/3731875478 >> v2:172.21.15.119:6800/1094129222 conn(0x7f6f4c06ede0 msgr2=0x7f6f4c071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.917+0000 7f6f62f26640 1 --2- 172.21.15.194:0/3731875478 >> v2:172.21.15.119:6800/1094129222 conn(0x7f6f4c06ede0 0x7f6f4c071290 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6f480094b0 tx=0x7f6f48035040 comp rx=0 tx=0).stop 2024-09-10T19:01:46.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.917+0000 7f6f62f26640 1 -- 172.21.15.194:0/3731875478 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f5c0fc820 msgr2=0x7f6f5c0724a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:46.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.917+0000 7f6f62f26640 1 --2- 172.21.15.194:0/3731875478 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f5c0fc820 0x7f6f5c0724a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6f5000c810 tx=0x7f6f500043c0 comp rx=0 tx=0).stop 2024-09-10T19:01:46.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.917+0000 7f6f62f26640 1 -- 172.21.15.194:0/3731875478 shutdown_connections 2024-09-10T19:01:46.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.917+0000 7f6f62f26640 1 --2- 172.21.15.194:0/3731875478 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6f5c0fe4a0 0x7f6f5c0710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:46.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.917+0000 7f6f62f26640 1 --2- 172.21.15.194:0/3731875478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6f5c0fdaf0 0x7f6f5c070b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:46.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.917+0000 7f6f62f26640 1 --2- 172.21.15.194:0/3731875478 >> v2:172.21.15.119:6800/1094129222 conn(0x7f6f4c06ede0 0x7f6f4c071290 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:46.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.917+0000 7f6f62f26640 1 --2- 172.21.15.194:0/3731875478 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f5c0fc820 0x7f6f5c0724a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:46.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.917+0000 7f6f62f26640 1 -- 172.21.15.194:0/3731875478 >> 172.21.15.194:0/3731875478 conn(0x7f6f5c0f7fc0 msgr2=0x7f6f5c0f9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:46.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.917+0000 7f6f62f26640 1 -- 172.21.15.194:0/3731875478 shutdown_connections 2024-09-10T19:01:46.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:46.917+0000 7f6f62f26640 1 -- 172.21.15.194:0/3731875478 wait complete. 2024-09-10T19:01:47.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:47 smithi194 ceph-mon[32464]: pgmap v181: 132 pgs: 18 creating+peering, 45 unknown, 69 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-10T19:01:47.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2161474448' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:01:47.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:47 smithi194 ceph-mon[32464]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:01:47.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:47 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:47.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1419775440' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T19:01:47.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:47 smithi194 ceph-mon[32464]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T19:01:47.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:47 smithi194 ceph-mon[32464]: osdmap e59: 8 total, 8 up, 8 in 2024-09-10T19:01:47.592 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-10T19:01:47.592 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T19:01:47.592 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-10T19:01:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:47 smithi119 ceph-mon[29577]: pgmap v181: 132 pgs: 18 creating+peering, 45 unknown, 69 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-10T19:01:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2161474448' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:01:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:47 smithi119 ceph-mon[29577]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:01:47.617 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:47 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:47.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1419775440' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T19:01:47.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:47 smithi119 ceph-mon[29577]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T19:01:47.618 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:47 smithi119 ceph-mon[29577]: osdmap e59: 8 total, 8 up, 8 in 2024-09-10T19:01:47.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:47 smithi119 ceph-mon[40674]: pgmap v181: 132 pgs: 18 creating+peering, 45 unknown, 69 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-09-10T19:01:47.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2161474448' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:01:47.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:47 smithi119 ceph-mon[40674]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:01:47.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:47 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:47.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1419775440' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T19:01:47.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:47 smithi119 ceph-mon[40674]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T19:01:47.618 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:47 smithi119 ceph-mon[40674]: osdmap e59: 8 total, 8 up, 8 in 2024-09-10T19:01:47.623 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-10T19:01:47.623 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-10T19:01:47.652 DEBUG:teuthology.orchestra.run.smithi194:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@iscsi.iscsi.a.service 2024-09-10T19:01:47.695 INFO:tasks.cephadm:Adding prometheus.a on smithi194 2024-09-10T19:01:47.695 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch apply prometheus '1;smithi194=a' 2024-09-10T19:01:48.675 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:01:48.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:48 smithi194 ceph-mon[32464]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.194", "placement": "1;smithi194=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:48.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:48 smithi194 ceph-mon[32464]: Saving service iscsi.datapool spec with placement smithi194=iscsi.a;count:1 2024-09-10T19:01:48.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:48 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:48.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:48 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:48.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:48 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:48.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:48 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-10T19:01:48.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:48 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-10T19:01:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:48 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:48 smithi194 ceph-mon[32464]: Deploying daemon iscsi.iscsi.a on smithi194 2024-09-10T19:01:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[29577]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.194", "placement": "1;smithi194=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[29577]: Saving service iscsi.datapool spec with placement smithi194=iscsi.a;count:1 2024-09-10T19:01:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-10T19:01:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-10T19:01:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[29577]: Deploying daemon iscsi.iscsi.a on smithi194 2024-09-10T19:01:48.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[40674]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.194", "placement": "1;smithi194=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[40674]: Saving service iscsi.datapool spec with placement smithi194=iscsi.a;count:1 2024-09-10T19:01:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-10T19:01:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-10T19:01:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:48 smithi119 ceph-mon[40674]: Deploying daemon iscsi.iscsi.a on smithi194 2024-09-10T19:01:49.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:49 smithi194 ceph-mon[32464]: pgmap v184: 132 pgs: 10 creating+peering, 18 unknown, 104 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 3.2 KiB/s wr, 60 op/s 2024-09-10T19:01:49.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:49 smithi194 ceph-mon[32464]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:01:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:49 smithi194 ceph-mon[32464]: Cluster is now healthy 2024-09-10T19:01:49.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:49 smithi119 ceph-mon[40674]: pgmap v184: 132 pgs: 10 creating+peering, 18 unknown, 104 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 3.2 KiB/s wr, 60 op/s 2024-09-10T19:01:49.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:49 smithi119 ceph-mon[40674]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:01:49.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:49 smithi119 ceph-mon[40674]: Cluster is now healthy 2024-09-10T19:01:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:49 smithi119 ceph-mon[29577]: pgmap v184: 132 pgs: 10 creating+peering, 18 unknown, 104 active+clean; 579 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 3.2 KiB/s wr, 60 op/s 2024-09-10T19:01:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:49 smithi119 ceph-mon[29577]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:01:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:49 smithi119 ceph-mon[29577]: Cluster is now healthy 2024-09-10T19:01:50.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.739+0000 7f4b40b27640 1 -- 172.21.15.194:0/887499849 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b3c0fb080 msgr2=0x7f4b3c104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:50.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.739+0000 7f4b40b27640 1 --2- 172.21.15.194:0/887499849 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b3c0fb080 0x7f4b3c104ec0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f4b34009cd0 tx=0x7f4b34004d10 comp rx=0 tx=0).stop 2024-09-10T19:01:50.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.739+0000 7f4b40b27640 1 -- 172.21.15.194:0/887499849 shutdown_connections 2024-09-10T19:01:50.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.739+0000 7f4b40b27640 1 --2- 172.21.15.194:0/887499849 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b3c0fb080 0x7f4b3c104ec0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:50.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.739+0000 7f4b40b27640 1 --2- 172.21.15.194:0/887499849 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b3c0fe7f0 0x7f4b3c0fa950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:50.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.739+0000 7f4b40b27640 1 --2- 172.21.15.194:0/887499849 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4b3c0fdea0 0x7f4b3c0fe2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:50.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.739+0000 7f4b40b27640 1 -- 172.21.15.194:0/887499849 >> 172.21.15.194:0/887499849 conn(0x7f4b3c0f66a0 msgr2=0x7f4b3c0f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:50.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.739+0000 7f4b40b27640 1 -- 172.21.15.194:0/887499849 shutdown_connections 2024-09-10T19:01:50.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.739+0000 7f4b40b27640 1 -- 172.21.15.194:0/887499849 wait complete. 2024-09-10T19:01:50.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b40b27640 1 Processor -- start 2024-09-10T19:01:50.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b40b27640 1 -- start start 2024-09-10T19:01:50.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b40b27640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4b3c0fb080 0x7f4b3c10cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:50.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b40b27640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b3c0fdea0 0x7f4b3c10d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:50.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b40b27640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b3c0fe7f0 0x7f4b3c114080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:50.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b40b27640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c114650 con 0x7f4b3c0fdea0 2024-09-10T19:01:50.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b40b27640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c1147c0 con 0x7f4b3c0fe7f0 2024-09-10T19:01:50.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b40b27640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3c114980 con 0x7f4b3c0fb080 2024-09-10T19:01:50.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b3ad9d640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b3c0fe7f0 0x7f4b3c114080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:50.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b3ad9d640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b3c0fe7f0 0x7f4b3c114080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:34852/0 (socket says 172.21.15.194:34852) 2024-09-10T19:01:50.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b3ad9d640 1 -- 172.21.15.194:0/3331263485 learned_addr learned my addr 172.21.15.194:0/3331263485 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T19:01:50.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b33fff640 1 --2- 172.21.15.194:0/3331263485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b3c0fdea0 0x7f4b3c10d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:50.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b3a59c640 1 --2- 172.21.15.194:0/3331263485 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4b3c0fb080 0x7f4b3c10cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:50.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b3ad9d640 1 -- 172.21.15.194:0/3331263485 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4b3c0fb080 msgr2=0x7f4b3c10cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:50.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b3ad9d640 1 --2- 172.21.15.194:0/3331263485 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4b3c0fb080 0x7f4b3c10cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:50.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b3ad9d640 1 -- 172.21.15.194:0/3331263485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b3c0fdea0 msgr2=0x7f4b3c10d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:50.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b3ad9d640 1 --2- 172.21.15.194:0/3331263485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b3c0fdea0 0x7f4b3c10d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:50.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b3ad9d640 1 -- 172.21.15.194:0/3331263485 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b340098b0 con 0x7f4b3c0fe7f0 2024-09-10T19:01:50.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.740+0000 7f4b3ad9d640 1 --2- 172.21.15.194:0/3331263485 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b3c0fe7f0 0x7f4b3c114080 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4b34004750 tx=0x7f4b34004d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:50.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.741+0000 7f4b32ffd640 1 -- 172.21.15.194:0/3331263485 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b34038070 con 0x7f4b3c0fe7f0 2024-09-10T19:01:50.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.741+0000 7f4b40b27640 1 -- 172.21.15.194:0/3331263485 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3c114c00 con 0x7f4b3c0fe7f0 2024-09-10T19:01:50.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.741+0000 7f4b32ffd640 1 -- 172.21.15.194:0/3331263485 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f4b34009730 con 0x7f4b3c0fe7f0 2024-09-10T19:01:50.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.741+0000 7f4b32ffd640 1 -- 172.21.15.194:0/3331263485 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3403cd70 con 0x7f4b3c0fe7f0 2024-09-10T19:01:50.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.741+0000 7f4b40b27640 1 -- 172.21.15.194:0/3331263485 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b3c1151a0 con 0x7f4b3c0fe7f0 2024-09-10T19:01:50.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.742+0000 7f4b40b27640 1 -- 172.21.15.194:0/3331263485 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3c04bf60 con 0x7f4b3c0fe7f0 2024-09-10T19:01:50.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.742+0000 7f4b32ffd640 1 -- 172.21.15.194:0/3331263485 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 15) v1 ==== 90746+0+0 (secure 0 0 0) 0x7f4b34030070 con 0x7f4b3c0fe7f0 2024-09-10T19:01:50.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.743+0000 7f4b32ffd640 1 --2- 172.21.15.194:0/3331263485 >> v2:172.21.15.119:6800/1094129222 conn(0x7f4b2806ee10 0x7f4b280712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:50.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.743+0000 7f4b32ffd640 1 -- 172.21.15.194:0/3331263485 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5938+0+0 (secure 0 0 0) 0x7f4b340aef60 con 0x7f4b3c0fe7f0 2024-09-10T19:01:50.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.743+0000 7f4b3a59c640 1 --2- 172.21.15.194:0/3331263485 >> v2:172.21.15.119:6800/1094129222 conn(0x7f4b2806ee10 0x7f4b280712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:50.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.743+0000 7f4b3a59c640 1 --2- 172.21.15.194:0/3331263485 >> v2:172.21.15.119:6800/1094129222 conn(0x7f4b2806ee10 0x7f4b280712c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c004b20 tx=0x7f4b2c009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:50.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.747+0000 7f4b32ffd640 1 -- 172.21.15.194:0/3331263485 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f4b3407bea0 con 0x7f4b3c0fe7f0 2024-09-10T19:01:50.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.923+0000 7f4b40b27640 1 -- 172.21.15.194:0/3331263485 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b3c107780 con 0x7f4b2806ee10 2024-09-10T19:01:50.934 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.933+0000 7f4b32ffd640 1 -- 172.21.15.194:0/3331263485 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f4b3c107780 con 0x7f4b2806ee10 2024-09-10T19:01:50.934 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled prometheus update... 2024-09-10T19:01:50.936 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.935+0000 7f4b40b27640 1 -- 172.21.15.194:0/3331263485 >> v2:172.21.15.119:6800/1094129222 conn(0x7f4b2806ee10 msgr2=0x7f4b280712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:50.936 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.935+0000 7f4b40b27640 1 --2- 172.21.15.194:0/3331263485 >> v2:172.21.15.119:6800/1094129222 conn(0x7f4b2806ee10 0x7f4b280712c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4b2c004b20 tx=0x7f4b2c009040 comp rx=0 tx=0).stop 2024-09-10T19:01:50.936 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.935+0000 7f4b40b27640 1 -- 172.21.15.194:0/3331263485 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b3c0fe7f0 msgr2=0x7f4b3c114080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:50.936 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.935+0000 7f4b40b27640 1 --2- 172.21.15.194:0/3331263485 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b3c0fe7f0 0x7f4b3c114080 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4b34004750 tx=0x7f4b34004d10 comp rx=0 tx=0).stop 2024-09-10T19:01:50.936 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.935+0000 7f4b40b27640 1 -- 172.21.15.194:0/3331263485 shutdown_connections 2024-09-10T19:01:50.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.935+0000 7f4b40b27640 1 --2- 172.21.15.194:0/3331263485 >> v2:172.21.15.119:6800/1094129222 conn(0x7f4b2806ee10 0x7f4b280712c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:50.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.935+0000 7f4b40b27640 1 --2- 172.21.15.194:0/3331263485 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b3c0fe7f0 0x7f4b3c114080 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:50.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.935+0000 7f4b40b27640 1 --2- 172.21.15.194:0/3331263485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b3c0fdea0 0x7f4b3c10d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:50.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.935+0000 7f4b40b27640 1 --2- 172.21.15.194:0/3331263485 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f4b3c0fb080 0x7f4b3c10cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:50.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.935+0000 7f4b40b27640 1 -- 172.21.15.194:0/3331263485 >> 172.21.15.194:0/3331263485 conn(0x7f4b3c0f66a0 msgr2=0x7f4b3c0f8190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:50.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.935+0000 7f4b40b27640 1 -- 172.21.15.194:0/3331263485 shutdown_connections 2024-09-10T19:01:50.937 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:50.935+0000 7f4b40b27640 1 -- 172.21.15.194:0/3331263485 wait complete. 2024-09-10T19:01:51.617 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:51 smithi194 ceph-mon[32464]: pgmap v185: 132 pgs: 2 creating+peering, 130 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 53 KiB/s rd, 4.8 KiB/s wr, 128 op/s 2024-09-10T19:01:51.620 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:51 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:51.620 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:51 smithi194 ceph-mon[32464]: osdmap e60: 8 total, 8 up, 8 in 2024-09-10T19:01:51.641 DEBUG:teuthology.orchestra.run.smithi194:prometheus.a> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@prometheus.a.service 2024-09-10T19:01:51.644 INFO:tasks.cephadm:Adding node-exporter.a on smithi119 2024-09-10T19:01:51.644 INFO:tasks.cephadm:Adding node-exporter.b on smithi194 2024-09-10T19:01:51.644 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch apply node-exporter '2;smithi119=a;smithi194=b' 2024-09-10T19:01:51.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:51 smithi119 ceph-mon[40674]: pgmap v185: 132 pgs: 2 creating+peering, 130 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 53 KiB/s rd, 4.8 KiB/s wr, 128 op/s 2024-09-10T19:01:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:51 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:51 smithi119 ceph-mon[40674]: osdmap e60: 8 total, 8 up, 8 in 2024-09-10T19:01:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:51 smithi119 ceph-mon[29577]: pgmap v185: 132 pgs: 2 creating+peering, 130 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 53 KiB/s rd, 4.8 KiB/s wr, 128 op/s 2024-09-10T19:01:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:51 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:51 smithi119 ceph-mon[29577]: osdmap e60: 8 total, 8 up, 8 in 2024-09-10T19:01:52.168 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 podman[61906]: 2024-09-10 19:01:52.078750985 +0000 UTC m=+0.139837607 container create 5fe7fd33d0dba93c2142639b5898165895b89af0b1aa38dd8338d92422d6ee15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T19:01:52.405 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:01:52.427 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 podman[61906]: 2024-09-10 19:01:52.29783185 +0000 UTC m=+0.358918474 container init 5fe7fd33d0dba93c2142639b5898165895b89af0b1aa38dd8338d92422d6ee15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T19:01:52.427 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 podman[61906]: 2024-09-10 19:01:52.302728557 +0000 UTC m=+0.363815182 container start 5fe7fd33d0dba93c2142639b5898165895b89af0b1aa38dd8338d92422d6ee15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-09-10T19:01:52.427 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 bash[61906]: 5fe7fd33d0dba93c2142639b5898165895b89af0b1aa38dd8338d92422d6ee15 2024-09-10T19:01:52.428 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 systemd[1]: Started Ceph iscsi.iscsi.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:01:52.696 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug Started the configuration object watcher 2024-09-10T19:01:52.697 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug Checking for config object changes every 1s 2024-09-10T19:01:52.697 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug Processing osd blocklist entries for this node 2024-09-10T19:01:52.697 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-mon[32464]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:52.697 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-mon[32464]: Saving service prometheus spec with placement smithi194=a;count:1 2024-09-10T19:01:52.697 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:52.697 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:52.697 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:52.698 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:52.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:52 smithi119 ceph-mon[29577]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:52 smithi119 ceph-mon[29577]: Saving service prometheus spec with placement smithi194=a;count:1 2024-09-10T19:01:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:52 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:52 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:52 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:52 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:52.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:52 smithi119 ceph-mon[40674]: from='client.24403 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:52.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:52 smithi119 ceph-mon[40674]: Saving service prometheus spec with placement smithi194=a;count:1 2024-09-10T19:01:52.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:52 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:52.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:52 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:52.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:52 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:52.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:52 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:53.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug Reading the configuration object to update local LIO configuration 2024-09-10T19:01:53.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug Configuration does not have an entry for this host(smithi194.front.sepia.ceph.com) - nothing to define to LIO 2024-09-10T19:01:53.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-10T19:01:53.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: * Environment: production 2024-09-10T19:01:53.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-10T19:01:53.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: Use a production WSGI server instead. 2024-09-10T19:01:53.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: * Debug mode: off 2024-09-10T19:01:53.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug * Running on all addresses. 2024-09-10T19:01:53.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-10T19:01:53.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: * Running on all addresses. 2024-09-10T19:01:53.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-10T19:01:53.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug * Running on http://[fe80::ec4:7aff:fe88:7ea5]:5000/ (Press CTRL+C to quit) 2024-09-10T19:01:53.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:01:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: * Running on http://[fe80::ec4:7aff:fe88:7ea5]:5000/ (Press CTRL+C to quit) 2024-09-10T19:01:53.704 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:53 smithi194 ceph-mon[32464]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 6.0 KiB/s wr, 184 op/s 2024-09-10T19:01:53.705 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:53 smithi194 ceph-mon[32464]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-10T19:01:53.705 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:53 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:53.705 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/3884510931' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-10T19:01:53.705 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:53 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:53.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:53 smithi119 ceph-mon[29577]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 6.0 KiB/s wr, 184 op/s 2024-09-10T19:01:53.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:53 smithi119 ceph-mon[29577]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-10T19:01:53.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:53 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:53.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/3884510931' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-10T19:01:53.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:53 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:53 smithi119 ceph-mon[40674]: pgmap v187: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 76 KiB/s rd, 6.0 KiB/s wr, 184 op/s 2024-09-10T19:01:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:53 smithi119 ceph-mon[40674]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-10T19:01:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:53 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:01:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/3884510931' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-10T19:01:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:53 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:53.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.987+0000 7fb929815640 1 -- 172.21.15.194:0/1466876583 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9240fd3e0 msgr2=0x7fb924103c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:53.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.987+0000 7fb929815640 1 --2- 172.21.15.194:0/1466876583 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9240fd3e0 0x7fb924103c80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb91c009cd0 tx=0x7fb91c004d10 comp rx=0 tx=0).stop 2024-09-10T19:01:53.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.987+0000 7fb929815640 1 -- 172.21.15.194:0/1466876583 shutdown_connections 2024-09-10T19:01:53.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.987+0000 7fb929815640 1 --2- 172.21.15.194:0/1466876583 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9240fd3e0 0x7fb924103c80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:53.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.987+0000 7fb929815640 1 --2- 172.21.15.194:0/1466876583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9240fca30 0x7fb9240fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:53.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.987+0000 7fb929815640 1 --2- 172.21.15.194:0/1466876583 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb9240fb760 0x7fb9240fbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:53.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.987+0000 7fb929815640 1 -- 172.21.15.194:0/1466876583 >> 172.21.15.194:0/1466876583 conn(0x7fb9240f6f20 msgr2=0x7fb9240f9350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:53.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.987+0000 7fb929815640 1 -- 172.21.15.194:0/1466876583 shutdown_connections 2024-09-10T19:01:53.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.987+0000 7fb929815640 1 -- 172.21.15.194:0/1466876583 wait complete. 2024-09-10T19:01:53.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.987+0000 7fb929815640 1 Processor -- start 2024-09-10T19:01:53.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.987+0000 7fb929815640 1 -- start start 2024-09-10T19:01:53.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb929815640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb9240fb760 0x7fb92410cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:53.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb929815640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9240fca30 0x7fb92410d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:53.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb929815640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9240fd3e0 0x7fb9241141a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:53.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb929815640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb924114770 con 0x7fb9240fd3e0 2024-09-10T19:01:53.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb929815640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9241148e0 con 0x7fb9240fca30 2024-09-10T19:01:53.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb929815640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb924114aa0 con 0x7fb9240fb760 2024-09-10T19:01:53.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb9227fc640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9240fca30 0x7fb92410d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:53.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb9227fc640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9240fca30 0x7fb92410d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:40550/0 (socket says 172.21.15.194:40550) 2024-09-10T19:01:53.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb9227fc640 1 -- 172.21.15.194:0/1779892741 learned_addr learned my addr 172.21.15.194:0/1779892741 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T19:01:53.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb922ffd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb9240fb760 0x7fb92410cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:53.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb9237fe640 1 --2- 172.21.15.194:0/1779892741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9240fd3e0 0x7fb9241141a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:53.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb9227fc640 1 -- 172.21.15.194:0/1779892741 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb9240fb760 msgr2=0x7fb92410cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:53.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb9227fc640 1 --2- 172.21.15.194:0/1779892741 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb9240fb760 0x7fb92410cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:53.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb9227fc640 1 -- 172.21.15.194:0/1779892741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9240fd3e0 msgr2=0x7fb9241141a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:53.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb9227fc640 1 --2- 172.21.15.194:0/1779892741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9240fd3e0 0x7fb9241141a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:53.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb9227fc640 1 -- 172.21.15.194:0/1779892741 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91c0098b0 con 0x7fb9240fca30 2024-09-10T19:01:53.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb9227fc640 1 --2- 172.21.15.194:0/1779892741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9240fca30 0x7fb92410d140 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb91800b810 tx=0x7fb9180091d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:53.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb90b7fe640 1 -- 172.21.15.194:0/1779892741 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91800be60 con 0x7fb9240fca30 2024-09-10T19:01:53.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb90b7fe640 1 -- 172.21.15.194:0/1779892741 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fb91800cba0 con 0x7fb9240fca30 2024-09-10T19:01:53.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb90b7fe640 1 -- 172.21.15.194:0/1779892741 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb918015b60 con 0x7fb9240fca30 2024-09-10T19:01:53.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.988+0000 7fb929815640 1 -- 172.21.15.194:0/1779892741 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb924114d80 con 0x7fb9240fca30 2024-09-10T19:01:53.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.989+0000 7fb929815640 1 -- 172.21.15.194:0/1779892741 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9241151f0 con 0x7fb9240fca30 2024-09-10T19:01:53.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.989+0000 7fb929815640 1 -- 172.21.15.194:0/1779892741 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92404bf60 con 0x7fb9240fca30 2024-09-10T19:01:53.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.990+0000 7fb90b7fe640 1 -- 172.21.15.194:0/1779892741 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 15) v1 ==== 90746+0+0 (secure 0 0 0) 0x7fb918012070 con 0x7fb9240fca30 2024-09-10T19:01:53.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.990+0000 7fb90b7fe640 1 --2- 172.21.15.194:0/1779892741 >> v2:172.21.15.119:6800/1094129222 conn(0x7fb90c06ee10 0x7fb90c0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:53.994 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.991+0000 7fb90b7fe640 1 -- 172.21.15.194:0/1779892741 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5938+0+0 (secure 0 0 0) 0x7fb91808d8b0 con 0x7fb9240fca30 2024-09-10T19:01:53.994 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.991+0000 7fb922ffd640 1 --2- 172.21.15.194:0/1779892741 >> v2:172.21.15.119:6800/1094129222 conn(0x7fb90c06ee10 0x7fb90c0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:53.994 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.991+0000 7fb922ffd640 1 --2- 172.21.15.194:0/1779892741 >> v2:172.21.15.119:6800/1094129222 conn(0x7fb90c06ee10 0x7fb90c0712c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb9140041d0 tx=0x7fb914004330 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:53.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:53.994+0000 7fb90b7fe640 1 -- 172.21.15.194:0/1779892741 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fb91805a920 con 0x7fb9240fca30 2024-09-10T19:01:54.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:54.175+0000 7fb929815640 1 -- 172.21.15.194:0/1779892741 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi119=a;smithi194=b", "target": ["mon-mgr", ""]}) v1 -- 0x7fb924106750 con 0x7fb90c06ee10 2024-09-10T19:01:54.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:54.182+0000 7fb90b7fe640 1 -- 172.21.15.194:0/1779892741 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fb924106750 con 0x7fb90c06ee10 2024-09-10T19:01:54.184 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled node-exporter update... 2024-09-10T19:01:54.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:54.184+0000 7fb929815640 1 -- 172.21.15.194:0/1779892741 >> v2:172.21.15.119:6800/1094129222 conn(0x7fb90c06ee10 msgr2=0x7fb90c0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:54.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:54.184+0000 7fb929815640 1 --2- 172.21.15.194:0/1779892741 >> v2:172.21.15.119:6800/1094129222 conn(0x7fb90c06ee10 0x7fb90c0712c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb9140041d0 tx=0x7fb914004330 comp rx=0 tx=0).stop 2024-09-10T19:01:54.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:54.184+0000 7fb929815640 1 -- 172.21.15.194:0/1779892741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9240fca30 msgr2=0x7fb92410d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:54.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:54.184+0000 7fb929815640 1 --2- 172.21.15.194:0/1779892741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9240fca30 0x7fb92410d140 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb91800b810 tx=0x7fb9180091d0 comp rx=0 tx=0).stop 2024-09-10T19:01:54.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:54.185+0000 7fb929815640 1 -- 172.21.15.194:0/1779892741 shutdown_connections 2024-09-10T19:01:54.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:54.185+0000 7fb929815640 1 --2- 172.21.15.194:0/1779892741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9240fd3e0 0x7fb9241141a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:54.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:54.185+0000 7fb929815640 1 --2- 172.21.15.194:0/1779892741 >> v2:172.21.15.119:6800/1094129222 conn(0x7fb90c06ee10 0x7fb90c0712c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:54.186 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:54.185+0000 7fb929815640 1 --2- 172.21.15.194:0/1779892741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9240fca30 0x7fb92410d140 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:54.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:54.185+0000 7fb929815640 1 --2- 172.21.15.194:0/1779892741 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb9240fb760 0x7fb92410cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:54.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:54.185+0000 7fb929815640 1 -- 172.21.15.194:0/1779892741 >> 172.21.15.194:0/1779892741 conn(0x7fb9240f6f20 msgr2=0x7fb9240f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:54.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:54.185+0000 7fb929815640 1 -- 172.21.15.194:0/1779892741 shutdown_connections 2024-09-10T19:01:54.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:54.185+0000 7fb929815640 1 -- 172.21.15.194:0/1779892741 wait complete. 2024-09-10T19:01:54.995 DEBUG:teuthology.orchestra.run.smithi119:node-exporter.a> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@node-exporter.a.service 2024-09-10T19:01:54.998 DEBUG:teuthology.orchestra.run.smithi194:node-exporter.b> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@node-exporter.b.service 2024-09-10T19:01:55.000 INFO:tasks.cephadm:Adding alertmanager.a on smithi119 2024-09-10T19:01:55.000 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch apply alertmanager '1;smithi119=a' 2024-09-10T19:01:55.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:55 smithi194 ceph-mon[32464]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi119=a;smithi194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:55.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:55 smithi194 ceph-mon[32464]: Saving service node-exporter spec with placement smithi119=a;smithi194=b;count:2 2024-09-10T19:01:55.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:55 smithi194 ceph-mon[32464]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 4.6 KiB/s wr, 143 op/s 2024-09-10T19:01:55.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:55 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:55.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:55 smithi119 ceph-mon[40674]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi119=a;smithi194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:55.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:55 smithi119 ceph-mon[40674]: Saving service node-exporter spec with placement smithi119=a;smithi194=b;count:2 2024-09-10T19:01:55.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:55 smithi119 ceph-mon[40674]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 4.6 KiB/s wr, 143 op/s 2024-09-10T19:01:55.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:55 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:55.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:55 smithi119 ceph-mon[29577]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi119=a;smithi194=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:55.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:55 smithi119 ceph-mon[29577]: Saving service node-exporter spec with placement smithi119=a;smithi194=b;count:2 2024-09-10T19:01:55.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:55 smithi119 ceph-mon[29577]: pgmap v188: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 59 KiB/s rd, 4.6 KiB/s wr, 143 op/s 2024-09-10T19:01:55.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:55 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:55.623 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:01:56.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:56 smithi194 ceph-mon[32464]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-10T19:01:56.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:56 smithi119 ceph-mon[29577]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-10T19:01:56.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:56 smithi119 ceph-mon[40674]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-10T19:01:57.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.125+0000 7f62eaf39640 1 -- 172.21.15.194:0/1583891726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e4066be0 msgr2=0x7f62e40ff4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:57.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.125+0000 7f62eaf39640 1 --2- 172.21.15.194:0/1583891726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e4066be0 0x7f62e40ff4a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f62e0004950 tx=0x7f62e0009da0 comp rx=0 tx=0).stop 2024-09-10T19:01:57.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.125+0000 7f62eaf39640 1 -- 172.21.15.194:0/1583891726 shutdown_connections 2024-09-10T19:01:57.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.125+0000 7f62eaf39640 1 --2- 172.21.15.194:0/1583891726 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62e41024d0 0x7f62e4104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:57.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.125+0000 7f62eaf39640 1 --2- 172.21.15.194:0/1583891726 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f62e40ff9e0 0x7f62e4101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:57.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.125+0000 7f62eaf39640 1 --2- 172.21.15.194:0/1583891726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e4066be0 0x7f62e40ff4a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:57.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.125+0000 7f62eaf39640 1 -- 172.21.15.194:0/1583891726 >> 172.21.15.194:0/1583891726 conn(0x7f62e40f6f10 msgr2=0x7f62e40f9340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:57.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.126+0000 7f62eaf39640 1 -- 172.21.15.194:0/1583891726 shutdown_connections 2024-09-10T19:01:57.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.126+0000 7f62eaf39640 1 -- 172.21.15.194:0/1583891726 wait complete. 2024-09-10T19:01:57.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.126+0000 7f62eaf39640 1 Processor -- start 2024-09-10T19:01:57.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.126+0000 7f62eaf39640 1 -- start start 2024-09-10T19:01:57.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.126+0000 7f62eaf39640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62e4066be0 0x7f62e410cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:57.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.126+0000 7f62eaf39640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f62e40ff9e0 0x7f62e410d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:57.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.126+0000 7f62eaf39640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e41024d0 0x7f62e4114330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:57.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.126+0000 7f62eaf39640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e4114900 con 0x7f62e4066be0 2024-09-10T19:01:57.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.126+0000 7f62eaf39640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e4114a70 con 0x7f62e41024d0 2024-09-10T19:01:57.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.126+0000 7f62eaf39640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62e4114c30 con 0x7f62e40ff9e0 2024-09-10T19:01:57.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.126+0000 7f62e94d6640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e41024d0 0x7f62e4114330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:57.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62e94d6640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e41024d0 0x7f62e4114330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:40578/0 (socket says 172.21.15.194:40578) 2024-09-10T19:01:57.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62e94d6640 1 -- 172.21.15.194:0/265016887 learned_addr learned my addr 172.21.15.194:0/265016887 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T19:01:57.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62e8cd5640 1 --2- 172.21.15.194:0/265016887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62e4066be0 0x7f62e410cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:57.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62dbfff640 1 --2- 172.21.15.194:0/265016887 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f62e40ff9e0 0x7f62e410d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:57.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62e94d6640 1 -- 172.21.15.194:0/265016887 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f62e40ff9e0 msgr2=0x7f62e410d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:57.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62e94d6640 1 --2- 172.21.15.194:0/265016887 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f62e40ff9e0 0x7f62e410d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:57.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62e94d6640 1 -- 172.21.15.194:0/265016887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62e4066be0 msgr2=0x7f62e410cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:57.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62e94d6640 1 --2- 172.21.15.194:0/265016887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62e4066be0 0x7f62e410cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:57.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62e94d6640 1 -- 172.21.15.194:0/265016887 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e0005040 con 0x7f62e41024d0 2024-09-10T19:01:57.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62e94d6640 1 --2- 172.21.15.194:0/265016887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e41024d0 0x7f62e4114330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f62dc00db80 tx=0x7f62dc004e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:57.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62d97fa640 1 -- 172.21.15.194:0/265016887 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc017070 con 0x7f62e41024d0 2024-09-10T19:01:57.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62eaf39640 1 -- 172.21.15.194:0/265016887 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e4114f10 con 0x7f62e41024d0 2024-09-10T19:01:57.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62d97fa640 1 -- 172.21.15.194:0/265016887 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f62dc00f040 con 0x7f62e41024d0 2024-09-10T19:01:57.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62d97fa640 1 -- 172.21.15.194:0/265016887 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62dc0036a0 con 0x7f62e41024d0 2024-09-10T19:01:57.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.127+0000 7f62eaf39640 1 -- 172.21.15.194:0/265016887 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e4115320 con 0x7f62e41024d0 2024-09-10T19:01:57.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.128+0000 7f62eaf39640 1 -- 172.21.15.194:0/265016887 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e404bf60 con 0x7f62e41024d0 2024-09-10T19:01:57.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.129+0000 7f62d97fa640 1 -- 172.21.15.194:0/265016887 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) v1 ==== 90797+0+0 (secure 0 0 0) 0x7f62dc00b890 con 0x7f62e41024d0 2024-09-10T19:01:57.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.129+0000 7f62d97fa640 1 --2- 172.21.15.194:0/265016887 >> v2:172.21.15.119:6800/1094129222 conn(0x7f62cc06ee70 0x7f62cc071320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:01:57.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.129+0000 7f62d97fa640 1 -- 172.21.15.194:0/265016887 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5938+0+0 (secure 0 0 0) 0x7f62dc08e380 con 0x7f62e41024d0 2024-09-10T19:01:57.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.129+0000 7f62e8cd5640 1 --2- 172.21.15.194:0/265016887 >> v2:172.21.15.119:6800/1094129222 conn(0x7f62cc06ee70 0x7f62cc071320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:01:57.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.130+0000 7f62e8cd5640 1 --2- 172.21.15.194:0/265016887 >> v2:172.21.15.119:6800/1094129222 conn(0x7f62cc06ee70 0x7f62cc071320 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f62e002d040 tx=0x7f62e0035040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:01:57.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.133+0000 7f62d97fa640 1 -- 172.21.15.194:0/265016887 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f62dc05b270 con 0x7f62e41024d0 2024-09-10T19:01:57.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:57 smithi194 ceph-mon[32464]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 3.8 KiB/s wr, 125 op/s 2024-09-10T19:01:57.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.308+0000 7f62eaf39640 1 -- 172.21.15.194:0/265016887 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f62e4107840 con 0x7f62cc06ee70 2024-09-10T19:01:57.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.314+0000 7f62d97fa640 1 -- 172.21.15.194:0/265016887 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f62e4107840 con 0x7f62cc06ee70 2024-09-10T19:01:57.317 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled alertmanager update... 2024-09-10T19:01:57.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.316+0000 7f62eaf39640 1 -- 172.21.15.194:0/265016887 >> v2:172.21.15.119:6800/1094129222 conn(0x7f62cc06ee70 msgr2=0x7f62cc071320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:57.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.316+0000 7f62eaf39640 1 --2- 172.21.15.194:0/265016887 >> v2:172.21.15.119:6800/1094129222 conn(0x7f62cc06ee70 0x7f62cc071320 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f62e002d040 tx=0x7f62e0035040 comp rx=0 tx=0).stop 2024-09-10T19:01:57.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.317+0000 7f62eaf39640 1 -- 172.21.15.194:0/265016887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e41024d0 msgr2=0x7f62e4114330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:01:57.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.317+0000 7f62eaf39640 1 --2- 172.21.15.194:0/265016887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e41024d0 0x7f62e4114330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f62dc00db80 tx=0x7f62dc004e60 comp rx=0 tx=0).stop 2024-09-10T19:01:57.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.317+0000 7f62eaf39640 1 -- 172.21.15.194:0/265016887 shutdown_connections 2024-09-10T19:01:57.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.317+0000 7f62eaf39640 1 --2- 172.21.15.194:0/265016887 >> v2:172.21.15.119:6800/1094129222 conn(0x7f62cc06ee70 0x7f62cc071320 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:57.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.317+0000 7f62eaf39640 1 --2- 172.21.15.194:0/265016887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62e41024d0 0x7f62e4114330 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:57.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.317+0000 7f62eaf39640 1 --2- 172.21.15.194:0/265016887 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f62e40ff9e0 0x7f62e410d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:57.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.317+0000 7f62eaf39640 1 --2- 172.21.15.194:0/265016887 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62e4066be0 0x7f62e410cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:01:57.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.317+0000 7f62eaf39640 1 -- 172.21.15.194:0/265016887 >> 172.21.15.194:0/265016887 conn(0x7f62e40f6f10 msgr2=0x7f62e40fdd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:01:57.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.317+0000 7f62eaf39640 1 -- 172.21.15.194:0/265016887 shutdown_connections 2024-09-10T19:01:57.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:01:57.317+0000 7f62eaf39640 1 -- 172.21.15.194:0/265016887 wait complete. 2024-09-10T19:01:57.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:57 smithi119 ceph-mon[29577]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 3.8 KiB/s wr, 125 op/s 2024-09-10T19:01:57.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:57 smithi119 ceph-mon[40674]: pgmap v189: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 52 KiB/s rd, 3.8 KiB/s wr, 125 op/s 2024-09-10T19:01:58.513 DEBUG:teuthology.orchestra.run.smithi119:alertmanager.a> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@alertmanager.a.service 2024-09-10T19:01:58.515 INFO:tasks.cephadm:Adding grafana.a on smithi194 2024-09-10T19:01:58.516 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph orch apply grafana '1;smithi194=a' 2024-09-10T19:01:58.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:58 smithi194 ceph-mon[32464]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:58.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:58 smithi194 ceph-mon[32464]: Saving service alertmanager spec with placement smithi119=a;count:1 2024-09-10T19:01:58.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:58 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:58.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:58 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:58.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:58 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:58.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:58 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:58.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:58 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:58.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:58 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:58.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:58 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:58.787 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:01:58.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[29577]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:58.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[29577]: Saving service alertmanager spec with placement smithi119=a;count:1 2024-09-10T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:58.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[40674]: from='client.24445 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi119=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:01:58.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[40674]: Saving service alertmanager spec with placement smithi119=a;count:1 2024-09-10T19:01:58.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:01:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:01:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:58 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:01:59.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:59 smithi194 ceph-mon[32464]: Deploying daemon prometheus.a on smithi194 2024-09-10T19:01:59.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:01:59 smithi194 ceph-mon[32464]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 2.7 KiB/s wr, 90 op/s 2024-09-10T19:01:59.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:59 smithi119 ceph-mon[29577]: Deploying daemon prometheus.a on smithi194 2024-09-10T19:01:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:01:59 smithi119 ceph-mon[29577]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 2.7 KiB/s wr, 90 op/s 2024-09-10T19:01:59.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:59 smithi119 ceph-mon[40674]: Deploying daemon prometheus.a on smithi194 2024-09-10T19:01:59.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:01:59 smithi119 ceph-mon[40674]: pgmap v190: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 2.7 KiB/s wr, 90 op/s 2024-09-10T19:02:00.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.782+0000 7f27d7dd7640 1 -- 172.21.15.194:0/653131743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27d0066c00 msgr2=0x7f27d00ff4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:00.830 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.782+0000 7f27d7dd7640 1 --2- 172.21.15.194:0/653131743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27d0066c00 0x7f27d00ff4d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f27cc004950 tx=0x7f27cc009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:00.830 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.782+0000 7f27d7dd7640 1 -- 172.21.15.194:0/653131743 shutdown_connections 2024-09-10T19:02:00.830 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.782+0000 7f27d7dd7640 1 --2- 172.21.15.194:0/653131743 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f27d01024b0 0x7f27d0104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:00.830 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.782+0000 7f27d7dd7640 1 --2- 172.21.15.194:0/653131743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27d00ffa10 0x7f27d0101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:00.830 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.782+0000 7f27d7dd7640 1 --2- 172.21.15.194:0/653131743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27d0066c00 0x7f27d00ff4d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:00.830 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.782+0000 7f27d7dd7640 1 -- 172.21.15.194:0/653131743 >> 172.21.15.194:0/653131743 conn(0x7f27d00f6f20 msgr2=0x7f27d00f9330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:00.830 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.782+0000 7f27d7dd7640 1 -- 172.21.15.194:0/653131743 shutdown_connections 2024-09-10T19:02:00.830 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.782+0000 7f27d7dd7640 1 -- 172.21.15.194:0/653131743 wait complete. 2024-09-10T19:02:00.830 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d7dd7640 1 Processor -- start 2024-09-10T19:02:00.830 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d7dd7640 1 -- start start 2024-09-10T19:02:00.830 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d7dd7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27d0066c00 0x7f27d010cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:00.830 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d7dd7640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27d00ffa10 0x7f27d010d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:00.830 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d7dd7640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f27d01024b0 0x7f27d0114320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d7dd7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d01148f0 con 0x7f27d0066c00 2024-09-10T19:02:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d7dd7640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d0114a60 con 0x7f27d00ffa10 2024-09-10T19:02:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d7dd7640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d0114c20 con 0x7f27d01024b0 2024-09-10T19:02:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d5372640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27d00ffa10 0x7f27d010d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d5372640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27d00ffa10 0x7f27d010d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:40584/0 (socket says 172.21.15.194:40584) 2024-09-10T19:02:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d5372640 1 -- 172.21.15.194:0/1179538500 learned_addr learned my addr 172.21.15.194:0/1179538500 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T19:02:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d6374640 1 --2- 172.21.15.194:0/1179538500 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f27d01024b0 0x7f27d0114320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d5b73640 1 --2- 172.21.15.194:0/1179538500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27d0066c00 0x7f27d010cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d5372640 1 -- 172.21.15.194:0/1179538500 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f27d01024b0 msgr2=0x7f27d0114320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d5372640 1 --2- 172.21.15.194:0/1179538500 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f27d01024b0 0x7f27d0114320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d5372640 1 -- 172.21.15.194:0/1179538500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27d0066c00 msgr2=0x7f27d010cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d5372640 1 --2- 172.21.15.194:0/1179538500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27d0066c00 0x7f27d010cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d5372640 1 -- 172.21.15.194:0/1179538500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc005040 con 0x7f27d00ffa10 2024-09-10T19:02:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d5b73640 1 --2- 172.21.15.194:0/1179538500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27d0066c00 0x7f27d010cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T19:02:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d5372640 1 --2- 172.21.15.194:0/1179538500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27d00ffa10 0x7f27d010d310 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f27c00028c0 tx=0x7f27c0004570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27c67fc640 1 -- 172.21.15.194:0/1179538500 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c0007c30 con 0x7f27d00ffa10 2024-09-10T19:02:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27c67fc640 1 -- 172.21.15.194:0/1179538500 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f27c000f040 con 0x7f27d00ffa10 2024-09-10T19:02:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27d7dd7640 1 -- 172.21.15.194:0/1179538500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d0114f00 con 0x7f27d00ffa10 2024-09-10T19:02:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.783+0000 7f27c67fc640 1 -- 172.21.15.194:0/1179538500 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c0009340 con 0x7f27d00ffa10 2024-09-10T19:02:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.784+0000 7f27d7dd7640 1 -- 172.21.15.194:0/1179538500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d01153c0 con 0x7f27d00ffa10 2024-09-10T19:02:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.785+0000 7f27d7dd7640 1 -- 172.21.15.194:0/1179538500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d004bf60 con 0x7f27d00ffa10 2024-09-10T19:02:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.785+0000 7f27c67fc640 1 -- 172.21.15.194:0/1179538500 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) v1 ==== 90797+0+0 (secure 0 0 0) 0x7f27c0009730 con 0x7f27d00ffa10 2024-09-10T19:02:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.786+0000 7f27c67fc640 1 --2- 172.21.15.194:0/1179538500 >> v2:172.21.15.119:6800/1094129222 conn(0x7f27bc06eea0 0x7f27bc071350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.786+0000 7f27c67fc640 1 -- 172.21.15.194:0/1179538500 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5938+0+0 (secure 0 0 0) 0x7f27c008e2e0 con 0x7f27d00ffa10 2024-09-10T19:02:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.786+0000 7f27d5b73640 1 --2- 172.21.15.194:0/1179538500 >> v2:172.21.15.119:6800/1094129222 conn(0x7f27bc06eea0 0x7f27bc071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:00.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.786+0000 7f27d5b73640 1 --2- 172.21.15.194:0/1179538500 >> v2:172.21.15.119:6800/1094129222 conn(0x7f27bc06eea0 0x7f27bc071350 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f27cc004920 tx=0x7f27cc035040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:00.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.790+0000 7f27c67fc640 1 -- 172.21.15.194:0/1179538500 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f27c005b120 con 0x7f27d00ffa10 2024-09-10T19:02:00.968 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.966+0000 7f27d7dd7640 1 -- 172.21.15.194:0/1179538500 --> v2:172.21.15.119:6800/1094129222 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}) v1 -- 0x7f27d0107800 con 0x7f27bc06eea0 2024-09-10T19:02:00.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.973+0000 7f27c67fc640 1 -- 172.21.15.194:0/1179538500 <== mgr.14150 v2:172.21.15.119:6800/1094129222 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f27d0107800 con 0x7f27bc06eea0 2024-09-10T19:02:00.974 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled grafana update... 2024-09-10T19:02:00.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.975+0000 7f27d7dd7640 1 -- 172.21.15.194:0/1179538500 >> v2:172.21.15.119:6800/1094129222 conn(0x7f27bc06eea0 msgr2=0x7f27bc071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:00.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.975+0000 7f27d7dd7640 1 --2- 172.21.15.194:0/1179538500 >> v2:172.21.15.119:6800/1094129222 conn(0x7f27bc06eea0 0x7f27bc071350 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f27cc004920 tx=0x7f27cc035040 comp rx=0 tx=0).stop 2024-09-10T19:02:00.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.975+0000 7f27d7dd7640 1 -- 172.21.15.194:0/1179538500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27d00ffa10 msgr2=0x7f27d010d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:00.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.975+0000 7f27d7dd7640 1 --2- 172.21.15.194:0/1179538500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27d00ffa10 0x7f27d010d310 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f27c00028c0 tx=0x7f27c0004570 comp rx=0 tx=0).stop 2024-09-10T19:02:00.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.975+0000 7f27d7dd7640 1 -- 172.21.15.194:0/1179538500 shutdown_connections 2024-09-10T19:02:00.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.975+0000 7f27d7dd7640 1 --2- 172.21.15.194:0/1179538500 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f27d01024b0 0x7f27d0114320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:00.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.975+0000 7f27d7dd7640 1 --2- 172.21.15.194:0/1179538500 >> v2:172.21.15.119:6800/1094129222 conn(0x7f27bc06eea0 0x7f27bc071350 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:00.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.975+0000 7f27d7dd7640 1 --2- 172.21.15.194:0/1179538500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f27d00ffa10 0x7f27d010d310 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:00.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.975+0000 7f27d7dd7640 1 --2- 172.21.15.194:0/1179538500 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27d0066c00 0x7f27d010cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:00.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.975+0000 7f27d7dd7640 1 -- 172.21.15.194:0/1179538500 >> 172.21.15.194:0/1179538500 conn(0x7f27d00f6f20 msgr2=0x7f27d00fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:00.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.975+0000 7f27d7dd7640 1 -- 172.21.15.194:0/1179538500 shutdown_connections 2024-09-10T19:02:00.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:00.975+0000 7f27d7dd7640 1 -- 172.21.15.194:0/1179538500 wait complete. 2024-09-10T19:02:01.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:01 smithi194 ceph-mon[32464]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 1.2 KiB/s wr, 40 op/s 2024-09-10T19:02:01.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:01 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:02:01.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:01 smithi119 ceph-mon[29577]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 1.2 KiB/s wr, 40 op/s 2024-09-10T19:02:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:01 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:02:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:01 smithi119 ceph-mon[40674]: pgmap v191: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 1.2 KiB/s wr, 40 op/s 2024-09-10T19:02:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:01 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:02:01.926 DEBUG:teuthology.orchestra.run.smithi194:grafana.a> sudo journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@grafana.a.service 2024-09-10T19:02:01.928 INFO:tasks.cephadm:Setting up client nodes... 2024-09-10T19:02:01.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-10T19:02:02.180 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:02.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:02 smithi119 ceph-mon[29577]: from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:02:02.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:02 smithi119 ceph-mon[29577]: Saving service grafana spec with placement smithi194=a;count:1 2024-09-10T19:02:02.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:02 smithi119 ceph-mon[40674]: from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:02:02.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:02 smithi119 ceph-mon[40674]: Saving service grafana spec with placement smithi194=a;count:1 2024-09-10T19:02:02.645 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:02 smithi194 ceph-mon[32464]: from='client.24451 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi194=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:02:02.645 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:02 smithi194 ceph-mon[32464]: Saving service grafana spec with placement smithi194=a;count:1 2024-09-10T19:02:03.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:02:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:02:03.481 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:03 smithi119 ceph-mon[29577]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 KiB/s wr, 33 op/s 2024-09-10T19:02:03.481 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:03 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:03.482 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:03 smithi119 ceph-mon[40674]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 KiB/s wr, 33 op/s 2024-09-10T19:02:03.482 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:03 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:03 smithi194 ceph-mon[32464]: pgmap v192: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1.0 KiB/s wr, 33 op/s 2024-09-10T19:02:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:03 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:04.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.012+0000 7f3082e9c640 1 -- 172.21.15.119:0/1843391961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f307c0fe800 msgr2=0x7f307c0fa960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.012+0000 7f3082e9c640 1 --2- 172.21.15.119:0/1843391961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f307c0fe800 0x7f307c0fa960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3068004980 tx=0x7f3068009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.012+0000 7f3082e9c640 1 -- 172.21.15.119:0/1843391961 shutdown_connections 2024-09-10T19:02:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.012+0000 7f3082e9c640 1 --2- 172.21.15.119:0/1843391961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f307c0fb090 0x7f307c104ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.012+0000 7f3082e9c640 1 --2- 172.21.15.119:0/1843391961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f307c0fe800 0x7f307c0fa960 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.012+0000 7f3082e9c640 1 --2- 172.21.15.119:0/1843391961 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f307c0fdeb0 0x7f307c0fe2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.012+0000 7f3082e9c640 1 -- 172.21.15.119:0/1843391961 >> 172.21.15.119:0/1843391961 conn(0x7f307c0f66f0 msgr2=0x7f307c0f8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.012+0000 7f3082e9c640 1 -- 172.21.15.119:0/1843391961 shutdown_connections 2024-09-10T19:02:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.012+0000 7f3082e9c640 1 -- 172.21.15.119:0/1843391961 wait complete. 2024-09-10T19:02:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.012+0000 7f3082e9c640 1 Processor -- start 2024-09-10T19:02:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.012+0000 7f3082e9c640 1 -- start start 2024-09-10T19:02:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3082e9c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f307c0fb090 0x7f307c0fd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3082e9c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f307c0fdeb0 0x7f307c0fbb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3082e9c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f307c0fe800 0x7f307c0fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3082e9c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c0fc810 con 0x7f307c0fe800 2024-09-10T19:02:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3082e9c640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f307c0fc980 con 0x7f307c0fdeb0 2024-09-10T19:02:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3082e9c640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f307c116e90 con 0x7f307c0fb090 2024-09-10T19:02:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3080c38640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f307c0fb090 0x7f307c0fd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3080c38640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f307c0fb090 0x7f307c0fd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:43272/0 (socket says 172.21.15.119:43272) 2024-09-10T19:02:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3081439640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f307c0fe800 0x7f307c0fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3080c38640 1 -- 172.21.15.119:0/668899460 learned_addr learned my addr 172.21.15.119:0/668899460 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f307bfff640 1 --2- 172.21.15.119:0/668899460 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f307c0fdeb0 0x7f307c0fbb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3080c38640 1 -- 172.21.15.119:0/668899460 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f307c0fdeb0 msgr2=0x7f307c0fbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3080c38640 1 --2- 172.21.15.119:0/668899460 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f307c0fdeb0 0x7f307c0fbb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3080c38640 1 -- 172.21.15.119:0/668899460 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f307c0fe800 msgr2=0x7f307c0fc0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3080c38640 1 --2- 172.21.15.119:0/668899460 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f307c0fe800 0x7f307c0fc0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3080c38640 1 -- 172.21.15.119:0/668899460 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3070005040 con 0x7f307c0fb090 2024-09-10T19:02:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3081439640 1 --2- 172.21.15.119:0/668899460 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f307c0fe800 0x7f307c0fc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:02:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3080c38640 1 --2- 172.21.15.119:0/668899460 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f307c0fb090 0x7f307c0fd4c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3070009780 tx=0x7f3070009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f30797fa640 1 -- 172.21.15.119:0/668899460 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30700099c0 con 0x7f307c0fb090 2024-09-10T19:02:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f3082e9c640 1 -- 172.21.15.119:0/668899460 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3068005040 con 0x7f307c0fb090 2024-09-10T19:02:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.013+0000 7f30797fa640 1 -- 172.21.15.119:0/668899460 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f307000f040 con 0x7f307c0fb090 2024-09-10T19:02:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.014+0000 7f30797fa640 1 -- 172.21.15.119:0/668899460 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3070013ca0 con 0x7f307c0fb090 2024-09-10T19:02:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.014+0000 7f3082e9c640 1 -- 172.21.15.119:0/668899460 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f307c1173c0 con 0x7f307c0fb090 2024-09-10T19:02:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.015+0000 7f3082e9c640 1 -- 172.21.15.119:0/668899460 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f307c107eb0 con 0x7f307c0fb090 2024-09-10T19:02:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.015+0000 7f30797fa640 1 -- 172.21.15.119:0/668899460 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 16) v1 ==== 90797+0+0 (secure 0 0 0) 0x7f307000ebb0 con 0x7f307c0fb090 2024-09-10T19:02:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.015+0000 7f30797fa640 1 --2- 172.21.15.119:0/668899460 >> v2:172.21.15.119:6800/1094129222 conn(0x7f306c06ee70 0x7f306c071320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:04.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.016+0000 7f307bfff640 1 --2- 172.21.15.119:0/668899460 >> v2:172.21.15.119:6800/1094129222 conn(0x7f306c06ee70 0x7f306c071320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:04.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.016+0000 7f30797fa640 1 -- 172.21.15.119:0/668899460 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5938+0+0 (secure 0 0 0) 0x7f307008e2c0 con 0x7f307c0fb090 2024-09-10T19:02:04.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.016+0000 7f307bfff640 1 --2- 172.21.15.119:0/668899460 >> v2:172.21.15.119:6800/1094129222 conn(0x7f306c06ee70 0x7f306c071320 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f306802d040 tx=0x7f3068035040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:04.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.020+0000 7f30797fa640 1 -- 172.21.15.119:0/668899460 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f307005b1b0 con 0x7f307c0fb090 2024-09-10T19:02:04.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.237+0000 7f3082e9c640 1 -- 172.21.15.119:0/668899460 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f307c103590 con 0x7f307c0fb090 2024-09-10T19:02:04.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.241+0000 7f30797fa640 1 -- 172.21.15.119:0/668899460 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v16) v1 ==== 170+0+59 (secure 0 0 0) 0x7f3070013e00 con 0x7f307c0fb090 2024-09-10T19:02:04.243 INFO:teuthology.orchestra.run.smithi119.stdout:[client.0] 2024-09-10T19:02:04.243 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQCsl+BmCilCDhAAxU+tOV5kigKy3QoAp3irYw== 2024-09-10T19:02:04.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.244+0000 7f3082e9c640 1 -- 172.21.15.119:0/668899460 >> v2:172.21.15.119:6800/1094129222 conn(0x7f306c06ee70 msgr2=0x7f306c071320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:04.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.244+0000 7f3082e9c640 1 --2- 172.21.15.119:0/668899460 >> v2:172.21.15.119:6800/1094129222 conn(0x7f306c06ee70 0x7f306c071320 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f306802d040 tx=0x7f3068035040 comp rx=0 tx=0).stop 2024-09-10T19:02:04.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.244+0000 7f3082e9c640 1 -- 172.21.15.119:0/668899460 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f307c0fb090 msgr2=0x7f307c0fd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:04.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.244+0000 7f3082e9c640 1 --2- 172.21.15.119:0/668899460 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f307c0fb090 0x7f307c0fd4c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3070009780 tx=0x7f3070009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:04.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.244+0000 7f3082e9c640 1 -- 172.21.15.119:0/668899460 shutdown_connections 2024-09-10T19:02:04.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.244+0000 7f3082e9c640 1 --2- 172.21.15.119:0/668899460 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f307c0fe800 0x7f307c0fc0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:04.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.244+0000 7f3082e9c640 1 --2- 172.21.15.119:0/668899460 >> v2:172.21.15.119:6800/1094129222 conn(0x7f306c06ee70 0x7f306c071320 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:04.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.244+0000 7f3082e9c640 1 --2- 172.21.15.119:0/668899460 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f307c0fdeb0 0x7f307c0fbb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:04.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.244+0000 7f3082e9c640 1 --2- 172.21.15.119:0/668899460 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f307c0fb090 0x7f307c0fd4c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:04.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.244+0000 7f3082e9c640 1 -- 172.21.15.119:0/668899460 >> 172.21.15.119:0/668899460 conn(0x7f307c0f66f0 msgr2=0x7f307c0f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:04.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.244+0000 7f3082e9c640 1 -- 172.21.15.119:0/668899460 shutdown_connections 2024-09-10T19:02:04.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:04.244+0000 7f3082e9c640 1 -- 172.21.15.119:0/668899460 wait complete. 2024-09-10T19:02:04.546 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/668899460' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:02:04.546 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:02:04.546 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T19:02:04.546 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/668899460' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:02:04.546 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:02:04.546 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T19:02:04.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/668899460' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:02:04.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:02:04.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T19:02:04.987 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T19:02:04.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-10T19:02:04.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-10T19:02:05.025 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-10T19:02:05.757 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:02:05.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:05 smithi194 ceph-mon[32464]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 85 B/s wr, 3 op/s 2024-09-10T19:02:05.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:05 smithi119 ceph-mon[29577]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 85 B/s wr, 3 op/s 2024-09-10T19:02:05.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:05 smithi119 ceph-mon[40674]: pgmap v193: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 85 B/s wr, 3 op/s 2024-09-10T19:02:07.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:07 smithi194 ceph-mon[32464]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-09-10T19:02:07.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:07 smithi119 ceph-mon[29577]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-09-10T19:02:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:07 smithi119 ceph-mon[40674]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-09-10T19:02:09.364 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:09 smithi194 ceph-mon[32464]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-09-10T19:02:09.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:09 smithi119 ceph-mon[29577]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-09-10T19:02:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:09 smithi119 ceph-mon[40674]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-09-10T19:02:10.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.701+0000 7fea02a73640 1 -- 172.21.15.194:0/3156155481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9fc0fdaf0 msgr2=0x7fe9fc0fdf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:10.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.701+0000 7fea02a73640 1 --2- 172.21.15.194:0/3156155481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9fc0fdaf0 0x7fe9fc0fdf60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4004950 tx=0x7fe9e4009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:10.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.701+0000 7fea02a73640 1 -- 172.21.15.194:0/3156155481 shutdown_connections 2024-09-10T19:02:10.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.701+0000 7fea02a73640 1 --2- 172.21.15.194:0/3156155481 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9fc0fe4a0 0x7fe9fc104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:10.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.701+0000 7fea02a73640 1 --2- 172.21.15.194:0/3156155481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9fc0fdaf0 0x7fe9fc0fdf60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:10.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.701+0000 7fea02a73640 1 --2- 172.21.15.194:0/3156155481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9fc0fc820 0x7fe9fc0fcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:10.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.701+0000 7fea02a73640 1 -- 172.21.15.194:0/3156155481 >> 172.21.15.194:0/3156155481 conn(0x7fe9fc0f7fc0 msgr2=0x7fe9fc0fa410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:10.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.701+0000 7fea02a73640 1 -- 172.21.15.194:0/3156155481 shutdown_connections 2024-09-10T19:02:10.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.701+0000 7fea02a73640 1 -- 172.21.15.194:0/3156155481 wait complete. 2024-09-10T19:02:10.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.701+0000 7fea02a73640 1 Processor -- start 2024-09-10T19:02:10.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.701+0000 7fea02a73640 1 -- start start 2024-09-10T19:02:10.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea02a73640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9fc0fc820 0x7fe9fc10ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:10.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea02a73640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9fc0fdaf0 0x7fe9fc10d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:10.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea02a73640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9fc0fe4a0 0x7fe9fc114450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:10.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea02a73640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc114a20 con 0x7fe9fc0fc820 2024-09-10T19:02:10.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea02a73640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc114b90 con 0x7fe9fc0fe4a0 2024-09-10T19:02:10.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea02a73640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc114d50 con 0x7fe9fc0fdaf0 2024-09-10T19:02:10.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea01010640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9fc0fe4a0 0x7fe9fc114450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:10.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea01010640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9fc0fe4a0 0x7fe9fc114450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:46966/0 (socket says 172.21.15.194:46966) 2024-09-10T19:02:10.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea01010640 1 -- 172.21.15.194:0/3928580430 learned_addr learned my addr 172.21.15.194:0/3928580430 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-09-10T19:02:10.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea0080f640 1 --2- 172.21.15.194:0/3928580430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9fc0fc820 0x7fe9fc10ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:10.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fe9fbfff640 1 --2- 172.21.15.194:0/3928580430 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9fc0fdaf0 0x7fe9fc10d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:10.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea01010640 1 -- 172.21.15.194:0/3928580430 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9fc0fdaf0 msgr2=0x7fe9fc10d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:10.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea01010640 1 --2- 172.21.15.194:0/3928580430 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9fc0fdaf0 0x7fe9fc10d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:10.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea01010640 1 -- 172.21.15.194:0/3928580430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9fc0fc820 msgr2=0x7fe9fc10ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:10.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea01010640 1 --2- 172.21.15.194:0/3928580430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9fc0fc820 0x7fe9fc10ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:10.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea01010640 1 -- 172.21.15.194:0/3928580430 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e4005040 con 0x7fe9fc0fe4a0 2024-09-10T19:02:10.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.702+0000 7fea01010640 1 --2- 172.21.15.194:0/3928580430 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9fc0fe4a0 0x7fe9fc114450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe9f400bc30 tx=0x7fe9f4009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:10.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.703+0000 7fe9f97fa640 1 -- 172.21.15.194:0/3928580430 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f40092e0 con 0x7fe9fc0fe4a0 2024-09-10T19:02:10.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.703+0000 7fea02a73640 1 -- 172.21.15.194:0/3928580430 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9fc114fd0 con 0x7fe9fc0fe4a0 2024-09-10T19:02:10.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.703+0000 7fe9f97fa640 1 -- 172.21.15.194:0/3928580430 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fe9f400ccd0 con 0x7fe9fc0fe4a0 2024-09-10T19:02:10.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.703+0000 7fe9f97fa640 1 -- 172.21.15.194:0/3928580430 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f4015d70 con 0x7fe9fc0fe4a0 2024-09-10T19:02:10.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.703+0000 7fea02a73640 1 -- 172.21.15.194:0/3928580430 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9fc1154c0 con 0x7fe9fc0fe4a0 2024-09-10T19:02:10.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.704+0000 7fea02a73640 1 -- 172.21.15.194:0/3928580430 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9fc107f10 con 0x7fe9fc0fe4a0 2024-09-10T19:02:10.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.704+0000 7fe9f97fa640 1 -- 172.21.15.194:0/3928580430 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) v1 ==== 90797+0+0 (secure 0 0 0) 0x7fe9f4012070 con 0x7fe9fc0fe4a0 2024-09-10T19:02:10.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.705+0000 7fe9f97fa640 1 --2- 172.21.15.194:0/3928580430 >> v2:172.21.15.119:6800/1094129222 conn(0x7fe9e806eea0 0x7fe9e8071350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:10.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.705+0000 7fe9f97fa640 1 -- 172.21.15.194:0/3928580430 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5938+0+0 (secure 0 0 0) 0x7fe9f408dfa0 con 0x7fe9fc0fe4a0 2024-09-10T19:02:10.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.705+0000 7fea0080f640 1 --2- 172.21.15.194:0/3928580430 >> v2:172.21.15.119:6800/1094129222 conn(0x7fe9e806eea0 0x7fe9e8071350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:10.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.705+0000 7fea0080f640 1 --2- 172.21.15.194:0/3928580430 >> v2:172.21.15.119:6800/1094129222 conn(0x7fe9e806eea0 0x7fe9e8071350 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009b50 tx=0x7fe9ec009120 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:10.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.709+0000 7fe9f97fa640 1 -- 172.21.15.194:0/3928580430 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fe9f405afa0 con 0x7fe9fc0fe4a0 2024-09-10T19:02:10.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.938+0000 7fea02a73640 1 -- 172.21.15.194:0/3928580430 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fe9fc103790 con 0x7fe9fc0fe4a0 2024-09-10T19:02:10.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.943+0000 7fe9f97fa640 1 -- 172.21.15.194:0/3928580430 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (secure 0 0 0) 0x7fe9f405a940 con 0x7fe9fc0fe4a0 2024-09-10T19:02:10.945 INFO:teuthology.orchestra.run.smithi194.stdout:[client.1] 2024-09-10T19:02:10.945 INFO:teuthology.orchestra.run.smithi194.stdout: key = AQCyl+BmzKYKOBAAGF8ks4e3ez86ybnMoC9JDg== 2024-09-10T19:02:10.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.946+0000 7fea02a73640 1 -- 172.21.15.194:0/3928580430 >> v2:172.21.15.119:6800/1094129222 conn(0x7fe9e806eea0 msgr2=0x7fe9e8071350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:10.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.946+0000 7fea02a73640 1 --2- 172.21.15.194:0/3928580430 >> v2:172.21.15.119:6800/1094129222 conn(0x7fe9e806eea0 0x7fe9e8071350 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009b50 tx=0x7fe9ec009120 comp rx=0 tx=0).stop 2024-09-10T19:02:10.947 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.946+0000 7fea02a73640 1 -- 172.21.15.194:0/3928580430 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9fc0fe4a0 msgr2=0x7fe9fc114450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:10.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.946+0000 7fea02a73640 1 --2- 172.21.15.194:0/3928580430 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9fc0fe4a0 0x7fe9fc114450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe9f400bc30 tx=0x7fe9f4009840 comp rx=0 tx=0).stop 2024-09-10T19:02:10.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.946+0000 7fea02a73640 1 -- 172.21.15.194:0/3928580430 shutdown_connections 2024-09-10T19:02:10.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.946+0000 7fea02a73640 1 --2- 172.21.15.194:0/3928580430 >> v2:172.21.15.119:6800/1094129222 conn(0x7fe9e806eea0 0x7fe9e8071350 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:10.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.946+0000 7fea02a73640 1 --2- 172.21.15.194:0/3928580430 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe9fc0fe4a0 0x7fe9fc114450 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:10.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.946+0000 7fea02a73640 1 --2- 172.21.15.194:0/3928580430 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe9fc0fdaf0 0x7fe9fc10d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:10.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.946+0000 7fea02a73640 1 --2- 172.21.15.194:0/3928580430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9fc0fc820 0x7fe9fc10ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:10.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.946+0000 7fea02a73640 1 -- 172.21.15.194:0/3928580430 >> 172.21.15.194:0/3928580430 conn(0x7fe9fc0f7fc0 msgr2=0x7fe9fc0f9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:10.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.946+0000 7fea02a73640 1 -- 172.21.15.194:0/3928580430 shutdown_connections 2024-09-10T19:02:10.948 INFO:teuthology.orchestra.run.smithi194.stderr:2024-09-10T19:02:10.946+0000 7fea02a73640 1 -- 172.21.15.194:0/3928580430 wait complete. 2024-09-10T19:02:11.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:11 smithi194 ceph-mon[32464]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:11.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.194:0/3928580430' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:02:11.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:02:11.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T19:02:11.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:11 smithi119 ceph-mon[29577]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.194:0/3928580430' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:02:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:02:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T19:02:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:11 smithi119 ceph-mon[40674]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.194:0/3928580430' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:02:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:02:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T19:02:11.836 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-09-10T19:02:11.836 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-10T19:02:11.836 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-10T19:02:11.877 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-10T19:02:11.877 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-10T19:02:11.877 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph mgr dump --format=json 2024-09-10T19:02:12.128 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:12.524 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:02:12.524 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:02:12.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:02:12.525 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-10T19:02:12.525 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:02:12.525 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:02:12.525 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:02:12.525 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-10T19:02:12.763 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:12 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:02:12.763 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:12 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:02:12.763 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:12 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' 2024-09-10T19:02:12.763 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:12 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-10T19:02:12.764 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ignoring --setuser ceph since I am not root 2024-09-10T19:02:12.764 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ignoring --setgroup ceph since I am not root 2024-09-10T19:02:12.764 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:12 smithi194 ceph-mgr[34573]: -- 172.21.15.194:0/3974176550 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555e85284340 con 0x555e85290c00 2024-09-10T19:02:12.764 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:12.644+0000 7f4f67c7fec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T19:02:12.764 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:12.698+0000 7f4f67c7fec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T19:02:12.764 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:12.761+0000 7f4f67c7fec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T19:02:12.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ignoring --setuser ceph since I am not root 2024-09-10T19:02:12.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ignoring --setgroup ceph since I am not root 2024-09-10T19:02:12.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-mgr[29850]: -- 172.21.15.119:0/691689914 <== mon.0 v2:172.21.15.119:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e407dec340 con 0x55e407df9000 2024-09-10T19:02:12.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:12.646+0000 7f07c2b1bec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T19:02:12.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:12.700+0000 7f07c2b1bec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T19:02:12.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:12.766+0000 7f07c2b1bec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T19:02:13.018 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:12.883+0000 7f4f67c7fec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:02:13.273 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:13 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:13.017+0000 7f4f67c7fec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T19:02:13.292 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:12.893+0000 7f07c2b1bec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:02:13.293 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:13 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:13.031+0000 7f07c2b1bec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T19:02:13.293 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:13 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:13.291+0000 7f07c2b1bec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T19:02:13.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:13 smithi119 ceph-mon[29577]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-10T19:02:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:13 smithi119 ceph-mon[29577]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-10T19:02:13.567 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:13 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:13.352+0000 7f07c2b1bec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T19:02:13.567 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:13 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:13.521+0000 7f07c2b1bec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T19:02:13.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:13 smithi119 ceph-mon[40674]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-10T19:02:13.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:13 smithi119 ceph-mon[40674]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-10T19:02:13.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:13 smithi194 ceph-mon[32464]: from='mgr.14150 172.21.15.119:0/53551783' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-10T19:02:13.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:13 smithi194 ceph-mon[32464]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-10T19:02:13.768 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:13 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:13.272+0000 7f4f67c7fec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T19:02:13.768 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:13 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:13.332+0000 7f4f67c7fec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T19:02:13.769 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:13 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:13.491+0000 7f4f67c7fec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T19:02:13.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.912+0000 7fddc7fb6640 1 -- 172.21.15.119:0/4083489326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddc00fb800 msgr2=0x7fddc00fbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:13.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.912+0000 7fddc7fb6640 1 --2- 172.21.15.119:0/4083489326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddc00fb800 0x7fddc00fbc70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fddb00048c0 tx=0x7fddb0009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:13.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.912+0000 7fddc7fb6640 1 -- 172.21.15.119:0/4083489326 shutdown_connections 2024-09-10T19:02:13.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.912+0000 7fddc7fb6640 1 --2- 172.21.15.119:0/4083489326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc00fc1b0 0x7fddc0108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:13.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.912+0000 7fddc7fb6640 1 --2- 172.21.15.119:0/4083489326 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddc00fb800 0x7fddc00fbc70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:13.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.912+0000 7fddc7fb6640 1 --2- 172.21.15.119:0/4083489326 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fddc01017d0 0x7fddc0101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:13.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.912+0000 7fddc7fb6640 1 -- 172.21.15.119:0/4083489326 >> 172.21.15.119:0/4083489326 conn(0x7fddc0071c00 msgr2=0x7fddc0072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:13.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.913+0000 7fddc7fb6640 1 -- 172.21.15.119:0/4083489326 shutdown_connections 2024-09-10T19:02:13.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.913+0000 7fddc7fb6640 1 -- 172.21.15.119:0/4083489326 wait complete. 2024-09-10T19:02:13.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.913+0000 7fddc7fb6640 1 Processor -- start 2024-09-10T19:02:13.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.913+0000 7fddc7fb6640 1 -- start start 2024-09-10T19:02:13.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.913+0000 7fddc7fb6640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fddc00fb800 0x7fddc00fd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:13.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.913+0000 7fddc7fb6640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc00fc1b0 0x7fddc00fd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:13.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.913+0000 7fddc7fb6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddc01017d0 0x7fddc00fde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:13.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.913+0000 7fddc7fb6640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc011b390 con 0x7fddc01017d0 2024-09-10T19:02:13.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.913+0000 7fddc7fb6640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddc011b500 con 0x7fddc00fc1b0 2024-09-10T19:02:13.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.913+0000 7fddc7fb6640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddc011b6c0 con 0x7fddc00fb800 2024-09-10T19:02:13.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.913+0000 7fddc5d52640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fddc00fb800 0x7fddc00fd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:13.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.913+0000 7fddc6553640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddc01017d0 0x7fddc00fde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:13.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddc6553640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddc01017d0 0x7fddc00fde40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51046/0 (socket says 172.21.15.119:51046) 2024-09-10T19:02:13.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddc5551640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc00fc1b0 0x7fddc00fd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:13.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddc5d52640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fddc00fb800 0x7fddc00fd3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:34454/0 (socket says 172.21.15.119:34454) 2024-09-10T19:02:13.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddc6553640 1 -- 172.21.15.119:0/1160823436 learned_addr learned my addr 172.21.15.119:0/1160823436 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:13.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddc5551640 1 -- 172.21.15.119:0/1160823436 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fddc00fb800 msgr2=0x7fddc00fd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:13.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddc5551640 1 --2- 172.21.15.119:0/1160823436 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fddc00fb800 0x7fddc00fd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:13.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddc5551640 1 -- 172.21.15.119:0/1160823436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddc01017d0 msgr2=0x7fddc00fde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:13.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddc5551640 1 --2- 172.21.15.119:0/1160823436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddc01017d0 0x7fddc00fde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:13.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddc5551640 1 -- 172.21.15.119:0/1160823436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddbc005040 con 0x7fddc00fc1b0 2024-09-10T19:02:13.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddc6553640 1 --2- 172.21.15.119:0/1160823436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddc01017d0 0x7fddc00fde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:02:13.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddc5d52640 1 --2- 172.21.15.119:0/1160823436 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fddc00fb800 0x7fddc00fd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:02:13.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddc5551640 1 --2- 172.21.15.119:0/1160823436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc00fc1b0 0x7fddc00fd900 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fddb002d040 tx=0x7fddb0035040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:13.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddb67fc640 1 -- 172.21.15.119:0/1160823436 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0007d00 con 0x7fddc00fc1b0 2024-09-10T19:02:13.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddb67fc640 1 -- 172.21.15.119:0/1160823436 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fddb00090d0 con 0x7fddc00fc1b0 2024-09-10T19:02:13.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddc7fb6640 1 -- 172.21.15.119:0/1160823436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb0005040 con 0x7fddc00fc1b0 2024-09-10T19:02:13.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddb67fc640 1 -- 172.21.15.119:0/1160823436 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddb0033e90 con 0x7fddc00fc1b0 2024-09-10T19:02:13.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.914+0000 7fddc7fb6640 1 -- 172.21.15.119:0/1160823436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddc011bd00 con 0x7fddc00fc1b0 2024-09-10T19:02:13.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.915+0000 7fddc7fb6640 1 -- 172.21.15.119:0/1160823436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddc00fcef0 con 0x7fddc00fc1b0 2024-09-10T19:02:13.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.916+0000 7fddb67fc640 1 -- 172.21.15.119:0/1160823436 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) v1 ==== 90811+0+0 (secure 0 0 0) 0x7fddb0030020 con 0x7fddc00fc1b0 2024-09-10T19:02:13.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.917+0000 7fddb67fc640 1 --2- 172.21.15.119:0/1160823436 >> v2:172.21.15.119:6800/1094129222 conn(0x7fddac06eec0 0x7fddac071370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:13.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.917+0000 7fddc5d52640 1 -- 172.21.15.119:0/1160823436 >> v2:172.21.15.119:6800/1094129222 conn(0x7fddac06eec0 msgr2=0x7fddac071370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1094129222 2024-09-10T19:02:13.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.917+0000 7fddb67fc640 1 -- 172.21.15.119:0/1160823436 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5938+0+0 (secure 0 0 0) 0x7fddb00adbf0 con 0x7fddc00fc1b0 2024-09-10T19:02:13.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.917+0000 7fddc5d52640 1 --2- 172.21.15.119:0/1160823436 >> v2:172.21.15.119:6800/1094129222 conn(0x7fddac06eec0 0x7fddac071370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T19:02:13.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:13.921+0000 7fddb67fc640 1 -- 172.21.15.119:0/1160823436 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fddb007ae10 con 0x7fddc00fc1b0 2024-09-10T19:02:14.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.118+0000 7fddc5d52640 1 -- 172.21.15.119:0/1160823436 >> v2:172.21.15.119:6800/1094129222 conn(0x7fddac06eec0 msgr2=0x7fddac071370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1094129222 2024-09-10T19:02:14.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.118+0000 7fddc5d52640 1 --2- 172.21.15.119:0/1160823436 >> v2:172.21.15.119:6800/1094129222 conn(0x7fddac06eec0 0x7fddac071370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T19:02:14.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.120+0000 7fddc7fb6640 1 -- 172.21.15.119:0/1160823436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fddc00fe8a0 con 0x7fddc00fc1b0 2024-09-10T19:02:14.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.123+0000 7fddb67fc640 1 -- 172.21.15.119:0/1160823436 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+176675 (secure 0 0 0) 0x7fddb0041020 con 0x7fddc00fc1b0 2024-09-10T19:02:14.128 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T19:02:14.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.126+0000 7fddc7fb6640 1 -- 172.21.15.119:0/1160823436 >> v2:172.21.15.119:6800/1094129222 conn(0x7fddac06eec0 msgr2=0x7fddac071370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T19:02:14.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.126+0000 7fddc7fb6640 1 --2- 172.21.15.119:0/1160823436 >> v2:172.21.15.119:6800/1094129222 conn(0x7fddac06eec0 0x7fddac071370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:14.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.126+0000 7fddc7fb6640 1 -- 172.21.15.119:0/1160823436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc00fc1b0 msgr2=0x7fddc00fd900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:14.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.126+0000 7fddc7fb6640 1 --2- 172.21.15.119:0/1160823436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc00fc1b0 0x7fddc00fd900 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fddb002d040 tx=0x7fddb0035040 comp rx=0 tx=0).stop 2024-09-10T19:02:14.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.126+0000 7fddc7fb6640 1 -- 172.21.15.119:0/1160823436 shutdown_connections 2024-09-10T19:02:14.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.126+0000 7fddc7fb6640 1 --2- 172.21.15.119:0/1160823436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fddc01017d0 0x7fddc00fde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:14.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.126+0000 7fddc7fb6640 1 --2- 172.21.15.119:0/1160823436 >> v2:172.21.15.119:6800/1094129222 conn(0x7fddac06eec0 0x7fddac071370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:14.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.126+0000 7fddc7fb6640 1 --2- 172.21.15.119:0/1160823436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc00fc1b0 0x7fddc00fd900 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:14.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.126+0000 7fddc7fb6640 1 --2- 172.21.15.119:0/1160823436 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fddc00fb800 0x7fddc00fd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:14.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.126+0000 7fddc7fb6640 1 -- 172.21.15.119:0/1160823436 >> 172.21.15.119:0/1160823436 conn(0x7fddc0071c00 msgr2=0x7fddc00653b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:14.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.126+0000 7fddc7fb6640 1 -- 172.21.15.119:0/1160823436 shutdown_connections 2024-09-10T19:02:14.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:14.126+0000 7fddc7fb6640 1 -- 172.21.15.119:0/1160823436 wait complete. 2024-09-10T19:02:14.728 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1160823436' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-10T19:02:14.729 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:14 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:14.399+0000 7f07c2b1bec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T19:02:14.729 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:14 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:14.464+0000 7f07c2b1bec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T19:02:14.729 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1160823436' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-10T19:02:14.755 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":1094129222}]},"active_addr":"172.21.15.119:6800/1094129222","active_change":"2024-09-10T18:57:18.128989+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24112,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.119:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":313114785}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":1352141759}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":3164939798}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":624043880}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.119:0","nonce":284317922}]}]} 2024-09-10T19:02:14.759 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-10T19:02:14.759 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-10T19:02:14.760 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T19:02:14.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1160823436' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-10T19:02:14.768 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:14 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:14.332+0000 7f4f67c7fec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T19:02:14.768 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:14 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:14.397+0000 7f4f67c7fec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T19:02:15.004 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:15.390 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:15.100+0000 7f4f67c7fec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T19:02:15.391 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:15.210+0000 7f4f67c7fec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T19:02:15.391 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:15.330+0000 7f4f67c7fec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:02:15.439 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:15 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:15.199+0000 7f07c2b1bec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T19:02:15.439 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:15 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:15.314+0000 7f07c2b1bec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T19:02:15.694 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:15 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:15.437+0000 7f07c2b1bec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:02:15.695 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:15 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:15.500+0000 7f07c2b1bec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T19:02:15.695 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:15 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:15.626+0000 7f07c2b1bec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T19:02:15.695 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:15 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:15.692+0000 7f07c2b1bec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T19:02:15.768 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:15.388+0000 7f4f67c7fec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T19:02:15.768 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:15.512+0000 7f4f67c7fec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T19:02:15.768 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:15.579+0000 7f4f67c7fec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T19:02:15.768 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:15.637+0000 7f4f67c7fec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T19:02:15.954 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:15 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:15.750+0000 7f07c2b1bec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T19:02:15.954 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:15 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:15.885+0000 7f07c2b1bec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T19:02:15.954 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:15 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:15.952+0000 7f07c2b1bec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T19:02:16.037 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:15.770+0000 7f4f67c7fec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T19:02:16.038 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:15.834+0000 7f4f67c7fec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T19:02:16.038 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:15.898+0000 7f4f67c7fec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T19:02:16.038 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:15.979+0000 7f4f67c7fec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T19:02:16.213 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:16.017+0000 7f07c2b1bec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T19:02:16.214 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:16.098+0000 7f07c2b1bec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T19:02:16.214 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:16.155+0000 7f07c2b1bec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T19:02:16.214 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:16.212+0000 7f07c2b1bec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T19:02:16.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:16.036+0000 7f4f67c7fec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T19:02:16.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:16.092+0000 7f4f67c7fec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T19:02:16.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.962+0000 7fe272437640 1 -- 172.21.15.119:0/2915572774 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe26c0fc1b0 msgr2=0x7fe26c108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:16.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.962+0000 7fe272437640 1 --2- 172.21.15.119:0/2915572774 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe26c0fc1b0 0x7fe26c108580 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe264009d00 tx=0x7fe264004d10 comp rx=0 tx=0).stop 2024-09-10T19:02:16.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.962+0000 7fe272437640 1 -- 172.21.15.119:0/2915572774 shutdown_connections 2024-09-10T19:02:16.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.962+0000 7fe272437640 1 --2- 172.21.15.119:0/2915572774 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe26c0fc1b0 0x7fe26c108580 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:16.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.962+0000 7fe272437640 1 --2- 172.21.15.119:0/2915572774 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe26c0fb800 0x7fe26c0fbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:16.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.962+0000 7fe272437640 1 --2- 172.21.15.119:0/2915572774 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe26c1017d0 0x7fe26c101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:16.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.962+0000 7fe272437640 1 -- 172.21.15.119:0/2915572774 >> 172.21.15.119:0/2915572774 conn(0x7fe26c071c00 msgr2=0x7fe26c072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:16.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.962+0000 7fe272437640 1 -- 172.21.15.119:0/2915572774 shutdown_connections 2024-09-10T19:02:16.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.962+0000 7fe272437640 1 -- 172.21.15.119:0/2915572774 wait complete. 2024-09-10T19:02:16.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe272437640 1 Processor -- start 2024-09-10T19:02:16.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe272437640 1 -- start start 2024-09-10T19:02:16.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe272437640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe26c0fb800 0x7fe26c112c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:16.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe272437640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe26c0fc1b0 0x7fe26c1131b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:16.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe272437640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe26c1017d0 0x7fe26c10ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:16.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe272437640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c10d460 con 0x7fe26c1017d0 2024-09-10T19:02:16.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe272437640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c10d5d0 con 0x7fe26c0fc1b0 2024-09-10T19:02:16.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe272437640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c10d790 con 0x7fe26c0fb800 2024-09-10T19:02:16.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe26bfff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe26c0fb800 0x7fe26c112c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:16.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe26bfff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe26c0fb800 0x7fe26c112c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:34464/0 (socket says 172.21.15.119:34464) 2024-09-10T19:02:16.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe26bfff640 1 -- 172.21.15.119:0/615482207 learned_addr learned my addr 172.21.15.119:0/615482207 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:16.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe2709d4640 1 --2- 172.21.15.119:0/615482207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe26c1017d0 0x7fe26c10ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:16.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe26b7fe640 1 --2- 172.21.15.119:0/615482207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe26c0fc1b0 0x7fe26c1131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:16.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe26bfff640 1 -- 172.21.15.119:0/615482207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe26c0fc1b0 msgr2=0x7fe26c1131b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:16.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe26bfff640 1 --2- 172.21.15.119:0/615482207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe26c0fc1b0 0x7fe26c1131b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:16.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe26bfff640 1 -- 172.21.15.119:0/615482207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe26c1017d0 msgr2=0x7fe26c10ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:17.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe26bfff640 1 --2- 172.21.15.119:0/615482207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe26c1017d0 0x7fe26c10ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:17.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.963+0000 7fe26bfff640 1 -- 172.21.15.119:0/615482207 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2640098b0 con 0x7fe26c0fb800 2024-09-10T19:02:17.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.964+0000 7fe26b7fe640 1 --2- 172.21.15.119:0/615482207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe26c0fc1b0 0x7fe26c1131b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T19:02:17.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.964+0000 7fe2709d4640 1 --2- 172.21.15.119:0/615482207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe26c1017d0 0x7fe26c10ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:02:17.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.964+0000 7fe26bfff640 1 --2- 172.21.15.119:0/615482207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe26c0fb800 0x7fe26c112c70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00ebf0 tx=0x7fe25c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:17.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.964+0000 7fe268ff9640 1 -- 172.21.15.119:0/615482207 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c007c30 con 0x7fe26c0fb800 2024-09-10T19:02:17.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.964+0000 7fe268ff9640 1 -- 172.21.15.119:0/615482207 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fe25c00f040 con 0x7fe26c0fb800 2024-09-10T19:02:17.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.964+0000 7fe272437640 1 -- 172.21.15.119:0/615482207 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c10da10 con 0x7fe26c0fb800 2024-09-10T19:02:17.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.964+0000 7fe268ff9640 1 -- 172.21.15.119:0/615482207 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c0094f0 con 0x7fe26c0fb800 2024-09-10T19:02:17.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.964+0000 7fe272437640 1 -- 172.21.15.119:0/615482207 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c119ca0 con 0x7fe26c0fb800 2024-09-10T19:02:17.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.965+0000 7fe272437640 1 -- 172.21.15.119:0/615482207 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c0fcef0 con 0x7fe26c0fb800 2024-09-10T19:02:17.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.966+0000 7fe268ff9640 1 -- 172.21.15.119:0/615482207 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 18) v1 ==== 90811+0+0 (secure 0 0 0) 0x7fe25c009860 con 0x7fe26c0fb800 2024-09-10T19:02:17.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.966+0000 7fe268ff9640 1 --2- 172.21.15.119:0/615482207 >> v2:172.21.15.119:6800/1094129222 conn(0x7fe25406eec0 0x7fe254071370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:17.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.966+0000 7fe26b7fe640 1 -- 172.21.15.119:0/615482207 >> v2:172.21.15.119:6800/1094129222 conn(0x7fe25406eec0 msgr2=0x7fe254071370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.119:6800/1094129222 2024-09-10T19:02:17.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.966+0000 7fe268ff9640 1 -- 172.21.15.119:0/615482207 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5938+0+0 (secure 0 0 0) 0x7fe25c08e120 con 0x7fe26c0fb800 2024-09-10T19:02:17.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.966+0000 7fe26b7fe640 1 --2- 172.21.15.119:0/615482207 >> v2:172.21.15.119:6800/1094129222 conn(0x7fe25406eec0 0x7fe254071370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T19:02:17.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:16.970+0000 7fe268ff9640 1 -- 172.21.15.119:0/615482207 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fe25c04e2a0 con 0x7fe26c0fb800 2024-09-10T19:02:17.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.019+0000 7fe268ff9640 1 -- 172.21.15.119:0/615482207 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mgrmap(e 19) v1 ==== 90462+0+0 (secure 0 0 0) 0x7fe25c056d10 con 0x7fe26c0fb800 2024-09-10T19:02:17.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.019+0000 7fe268ff9640 1 -- 172.21.15.119:0/615482207 >> v2:172.21.15.119:6800/1094129222 conn(0x7fe25406eec0 msgr2=0x7fe254071370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T19:02:17.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.019+0000 7fe268ff9640 1 --2- 172.21.15.119:0/615482207 >> v2:172.21.15.119:6800/1094129222 conn(0x7fe25406eec0 0x7fe254071370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:17.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.183+0000 7fe272437640 1 -- 172.21.15.119:0/615482207 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe26c10bda0 con 0x7fe26c0fb800 2024-09-10T19:02:17.185 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:17.007+0000 7f07c2b1bec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T19:02:17.185 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:02:17] ENGINE Bus STARTING 2024-09-10T19:02:17.185 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: CherryPy Checker: 2024-09-10T19:02:17.185 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: The Application mounted at '' has an empty config. 2024-09-10T19:02:17.185 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:17.186 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-mon[40674]: Standby manager daemon x restarted 2024-09-10T19:02:17.186 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-mon[40674]: Standby manager daemon x started 2024-09-10T19:02:17.186 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-mon[40674]: from='mgr.? 172.21.15.194:0/2726418838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T19:02:17.186 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-mon[40674]: from='mgr.? 172.21.15.194:0/2726418838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T19:02:17.186 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-mon[40674]: from='mgr.? 172.21.15.194:0/2726418838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T19:02:17.186 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-mon[40674]: from='mgr.? 172.21.15.194:0/2726418838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T19:02:17.186 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-mon[29577]: Standby manager daemon x restarted 2024-09-10T19:02:17.186 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-mon[29577]: Standby manager daemon x started 2024-09-10T19:02:17.186 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-mon[29577]: from='mgr.? 172.21.15.194:0/2726418838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T19:02:17.187 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-mon[29577]: from='mgr.? 172.21.15.194:0/2726418838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T19:02:17.187 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-mon[29577]: from='mgr.? 172.21.15.194:0/2726418838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T19:02:17.187 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:16 smithi119 ceph-mon[29577]: from='mgr.? 172.21.15.194:0/2726418838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T19:02:17.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.186+0000 7fe268ff9640 1 -- 172.21.15.119:0/615482207 <== mon.2 v2:172.21.15.119:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+19257 (secure 0 0 0) 0x7fe25c007d90 con 0x7fe26c0fb800 2024-09-10T19:02:17.189 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T19:02:17.189 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":61,"fsid":"3afe9804-6fa6-11ef-bcea-c7b262605968","created":"2024-09-10T18:56:26.299359+0000","modified":"2024-09-10T19:02:17.010185+0000","last_up_change":"2024-09-10T19:01:24.140998+0000","last_in_change":"2024-09-10T19:01:07.206749+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T18:59:30.453601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-10T19:01:39.009038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-10T19:01:39.241355+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-10T19:01:40.613730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-10T19:01:42.567074+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-10T19:01:44.643080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"7c2033fa-42d6-44b1-ac93-8e4ea2aea00f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6801","nonce":1598455131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1598455131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1598455131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6803","nonce":1598455131}]},"public_addr":"172.21.15.119:6801/1598455131","cluster_addr":"172.21.15.119:6802/1598455131","heartbeat_back_addr":"172.21.15.119:6804/1598455131","heartbeat_front_addr":"172.21.15.119:6803/1598455131","state":["exists","up"]},{"osd":1,"uuid":"cdc97929-345b-440d-9d0f-c64e3cbe8110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6805","nonce":4018071000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":4018071000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":4018071000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6807","nonce":4018071000}]},"public_addr":"172.21.15.119:6805/4018071000","cluster_addr":"172.21.15.119:6806/4018071000","heartbeat_back_addr":"172.21.15.119:6808/4018071000","heartbeat_front_addr":"172.21.15.119:6807/4018071000","state":["exists","up"]},{"osd":2,"uuid":"a53d3e5b-f03d-4082-a6d5-af183c1e8ca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6809","nonce":150641920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":150641920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":150641920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6811","nonce":150641920}]},"public_addr":"172.21.15.119:6809/150641920","cluster_addr":"172.21.15.119:6810/150641920","heartbeat_back_addr":"172.21.15.119:6812/150641920","heartbeat_front_addr":"172.21.15.119:6811/150641920","state":["exists","up"]},{"osd":3,"uuid":"291e81da-6fc3-44a4-880c-dd6c0ad00c13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6813","nonce":783850966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":783850966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":783850966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6815","nonce":783850966}]},"public_addr":"172.21.15.119:6813/783850966","cluster_addr":"172.21.15.119:6814/783850966","heartbeat_back_addr":"172.21.15.119:6816/783850966","heartbeat_front_addr":"172.21.15.119:6815/783850966","state":["exists","up"]},{"osd":4,"uuid":"0bee9361-6ab4-4545-a6d4-b05bbb4f9c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":4259870591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6801","nonce":4259870591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6803","nonce":4259870591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":4259870591}]},"public_addr":"172.21.15.194:6800/4259870591","cluster_addr":"172.21.15.194:6801/4259870591","heartbeat_back_addr":"172.21.15.194:6803/4259870591","heartbeat_front_addr":"172.21.15.194:6802/4259870591","state":["exists","up"]},{"osd":5,"uuid":"b26af90f-5d90-4929-94e6-92545dffd78d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":635822671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6805","nonce":635822671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6807","nonce":635822671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":635822671}]},"public_addr":"172.21.15.194:6804/635822671","cluster_addr":"172.21.15.194:6805/635822671","heartbeat_back_addr":"172.21.15.194:6807/635822671","heartbeat_front_addr":"172.21.15.194:6806/635822671","state":["exists","up"]},{"osd":6,"uuid":"c05bb66f-8bc0-44b3-965d-fda246401108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2830881352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6809","nonce":2830881352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6811","nonce":2830881352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2830881352}]},"public_addr":"172.21.15.194:6808/2830881352","cluster_addr":"172.21.15.194:6809/2830881352","heartbeat_back_addr":"172.21.15.194:6811/2830881352","heartbeat_front_addr":"172.21.15.194:6810/2830881352","state":["exists","up"]},{"osd":7,"uuid":"16049fee-bcde-450e-b4c5-de13dbb392d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":117762113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6813","nonce":117762113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6815","nonce":117762113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":117762113}]},"public_addr":"172.21.15.194:6812/117762113","cluster_addr":"172.21.15.194:6813/117762113","heartbeat_back_addr":"172.21.15.194:6815/117762113","heartbeat_front_addr":"172.21.15.194:6814/117762113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:58:39.672796+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:59:03.878834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:59:27.525213+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:59:51.755950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:00:12.576692+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:00:35.773984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:00:58.811225+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:01:22.187718+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/284317922":"2024-09-11T19:02:17.010143+0000","172.21.15.119:6800/4184327284":"2024-09-11T18:56:55.126452+0000","172.21.15.119:0/4285164139":"2024-09-11T18:56:55.126452+0000","172.21.15.119:0/3881701229":"2024-09-11T18:56:55.126452+0000","172.21.15.119:0/313114785":"2024-09-11T19:02:17.010143+0000","172.21.15.119:0/359148006":"2024-09-11T18:56:55.126452+0000","172.21.15.119:0/1889495967":"2024-09-11T18:57:18.128849+0000","172.21.15.119:6800/1136125279":"2024-09-11T18:57:18.128849+0000","172.21.15.119:6800/1094129222":"2024-09-11T19:02:17.010143+0000","172.21.15.119:0/651881751":"2024-09-11T18:57:18.128849+0000","172.21.15.119:0/667233824":"2024-09-11T18:57:18.128849+0000","172.21.15.119:0/3164939798":"2024-09-11T19:02:17.010143+0000","172.21.15.119:0/624043880":"2024-09-11T19:02:17.010143+0000","172.21.15.119:0/1352141759":"2024-09-11T19:02:17.010143+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:02:17.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.189+0000 7fe272437640 1 -- 172.21.15.119:0/615482207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe26c0fb800 msgr2=0x7fe26c112c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:17.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.189+0000 7fe272437640 1 --2- 172.21.15.119:0/615482207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe26c0fb800 0x7fe26c112c70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe25c00ebf0 tx=0x7fe25c00b040 comp rx=0 tx=0).stop 2024-09-10T19:02:17.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.189+0000 7fe272437640 1 -- 172.21.15.119:0/615482207 shutdown_connections 2024-09-10T19:02:17.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.189+0000 7fe272437640 1 --2- 172.21.15.119:0/615482207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe26c1017d0 0x7fe26c10ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:17.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.189+0000 7fe272437640 1 --2- 172.21.15.119:0/615482207 >> v2:172.21.15.119:6800/1094129222 conn(0x7fe25406eec0 0x7fe254071370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:17.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.189+0000 7fe272437640 1 --2- 172.21.15.119:0/615482207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe26c0fc1b0 0x7fe26c1131b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:17.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.189+0000 7fe272437640 1 --2- 172.21.15.119:0/615482207 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe26c0fb800 0x7fe26c112c70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:17.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.189+0000 7fe272437640 1 -- 172.21.15.119:0/615482207 >> 172.21.15.119:0/615482207 conn(0x7fe26c071c00 msgr2=0x7fe26c0f9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:17.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.189+0000 7fe272437640 1 -- 172.21.15.119:0/615482207 shutdown_connections 2024-09-10T19:02:17.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:17.189+0000 7fe272437640 1 -- 172.21.15.119:0/615482207 wait complete. 2024-09-10T19:02:17.193 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-mon[32464]: Standby manager daemon x restarted 2024-09-10T19:02:17.193 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-mon[32464]: Standby manager daemon x started 2024-09-10T19:02:17.193 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-mon[32464]: from='mgr.? 172.21.15.194:0/2726418838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T19:02:17.193 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-mon[32464]: from='mgr.? 172.21.15.194:0/2726418838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T19:02:17.193 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-mon[32464]: from='mgr.? 172.21.15.194:0/2726418838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T19:02:17.193 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-mon[32464]: from='mgr.? 172.21.15.194:0/2726418838' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T19:02:17.194 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:16.829+0000 7f4f67c7fec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T19:02:17.194 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: [10/Sep/2024:19:02:16] ENGINE Bus STARTING 2024-09-10T19:02:17.194 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: CherryPy Checker: 2024-09-10T19:02:17.194 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: The Application mounted at '' has an empty config. 2024-09-10T19:02:17.194 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: 2024-09-10T19:02:17.194 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: [10/Sep/2024:19:02:16] ENGINE Serving on http://:::9283 2024-09-10T19:02:17.194 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: [10/Sep/2024:19:02:16] ENGINE Bus STARTED 2024-09-10T19:02:17.458 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:02:17] ENGINE Serving on http://:::9283 2024-09-10T19:02:17.458 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:02:17] ENGINE Bus STARTED 2024-09-10T19:02:18.009 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-10T19:02:18.009 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T19:02:18.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:02:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:02:18.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-10T19:02:18.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: Active manager daemon y restarted 2024-09-10T19:02:18.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: Activating manager daemon y 2024-09-10T19:02:18.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: osdmap e61: 8 total, 8 up, 8 in 2024-09-10T19:02:18.182 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: mgrmap e19: y(active, starting, since 0.0107693s), standbys: x 2024-09-10T19:02:18.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:02:18.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:02:18.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:02:18.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T19:02:18.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T19:02:18.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:02:18.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:02:18.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:02:18.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:02:18.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:02:18.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:02:18.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:02:18.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:02:18.183 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T19:02:18.184 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T19:02:18.184 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T19:02:18.184 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: Manager daemon y is now available 2024-09-10T19:02:18.184 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:02:18.184 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:18.184 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:18.184 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T19:02:18.184 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T19:02:18.184 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/615482207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:02:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-10T19:02:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: Active manager daemon y restarted 2024-09-10T19:02:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: Activating manager daemon y 2024-09-10T19:02:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: osdmap e61: 8 total, 8 up, 8 in 2024-09-10T19:02:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: mgrmap e19: y(active, starting, since 0.0107693s), standbys: x 2024-09-10T19:02:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:02:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:02:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:02:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T19:02:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T19:02:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: Manager daemon y is now available 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:18.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T19:02:18.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T19:02:18.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/615482207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:02:18.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-10T19:02:18.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: Active manager daemon y restarted 2024-09-10T19:02:18.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: Activating manager daemon y 2024-09-10T19:02:18.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: osdmap e61: 8 total, 8 up, 8 in 2024-09-10T19:02:18.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: mgrmap e19: y(active, starting, since 0.0107693s), standbys: x 2024-09-10T19:02:18.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:02:18.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:02:18.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:02:18.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T19:02:18.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: Manager daemon y is now available 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:18.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:18.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T19:02:18.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T19:02:18.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/615482207' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:02:18.594 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:18.635 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:18.379+0000 7f076589d640 -1 log_channel(cephadm) log [ERR] : [10/Sep/2024:19:02:18] ENGINE Error in HTTPServer.serve 2024-09-10T19:02:18.635 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: Traceback (most recent call last): 2024-09-10T19:02:18.635 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T19:02:18.636 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: self._connections.run(self.expiration_interval) 2024-09-10T19:02:18.636 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T19:02:18.636 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: self._run(expiration_interval) 2024-09-10T19:02:18.636 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T19:02:18.636 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T19:02:18.636 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T19:02:18.636 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T19:02:18.636 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T19:02:18.636 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: s = self.context.wrap_socket( 2024-09-10T19:02:18.636 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T19:02:18.636 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: return self.sslsocket_class._create( 2024-09-10T19:02:18.636 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T19:02:18.636 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: self.do_handshake() 2024-09-10T19:02:18.637 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T19:02:18.637 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: self._sslobj.do_handshake() 2024-09-10T19:02:18.637 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T19:02:18.637 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:02:19.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:19 smithi194 ceph-mon[32464]: mgrmap e20: y(active, since 1.01857s), standbys: x 2024-09-10T19:02:19.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:19 smithi194 ceph-mon[32464]: [10/Sep/2024:19:02:18] ENGINE Bus STARTING 2024-09-10T19:02:19.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:19 smithi194 ceph-mon[32464]: [10/Sep/2024:19:02:18] ENGINE Serving on https://172.21.15.119:7150 2024-09-10T19:02:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:19 smithi194 ceph-mon[32464]: [10/Sep/2024:19:02:18] ENGINE Bus STARTED 2024-09-10T19:02:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:19 smithi194 ceph-mon[32464]: [10/Sep/2024:19:02:18] ENGINE Error in HTTPServer.serve 2024-09-10T19:02:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout: Traceback (most recent call last): 2024-09-10T19:02:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T19:02:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout: self._connections.run(self.expiration_interval) 2024-09-10T19:02:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T19:02:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout: self._run(expiration_interval) 2024-09-10T19:02:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T19:02:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T19:02:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T19:02:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T19:02:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T19:02:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout: s = self.context.wrap_socket( 2024-09-10T19:02:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T19:02:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout: return self.sslsocket_class._create( 2024-09-10T19:02:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T19:02:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout: self.do_handshake() 2024-09-10T19:02:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T19:02:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout: self._sslobj.do_handshake() 2024-09-10T19:02:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T19:02:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:19 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:19.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:19 smithi119 ceph-mon[40674]: mgrmap e20: y(active, since 1.01857s), standbys: x 2024-09-10T19:02:19.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:19 smithi119 ceph-mon[40674]: [10/Sep/2024:19:02:18] ENGINE Bus STARTING 2024-09-10T19:02:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:19 smithi119 ceph-mon[40674]: [10/Sep/2024:19:02:18] ENGINE Serving on https://172.21.15.119:7150 2024-09-10T19:02:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:19 smithi119 ceph-mon[40674]: [10/Sep/2024:19:02:18] ENGINE Bus STARTED 2024-09-10T19:02:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:19 smithi119 ceph-mon[40674]: [10/Sep/2024:19:02:18] ENGINE Error in HTTPServer.serve 2024-09-10T19:02:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout: Traceback (most recent call last): 2024-09-10T19:02:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T19:02:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._connections.run(self.expiration_interval) 2024-09-10T19:02:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T19:02:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._run(expiration_interval) 2024-09-10T19:02:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T19:02:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T19:02:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T19:02:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T19:02:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T19:02:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout: s = self.context.wrap_socket( 2024-09-10T19:02:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T19:02:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout: return self.sslsocket_class._create( 2024-09-10T19:02:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T19:02:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout: self.do_handshake() 2024-09-10T19:02:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T19:02:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout: self._sslobj.do_handshake() 2024-09-10T19:02:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T19:02:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:19 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:19.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:19 smithi119 ceph-mon[29577]: mgrmap e20: y(active, since 1.01857s), standbys: x 2024-09-10T19:02:19.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:19 smithi119 ceph-mon[29577]: [10/Sep/2024:19:02:18] ENGINE Bus STARTING 2024-09-10T19:02:19.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:19 smithi119 ceph-mon[29577]: [10/Sep/2024:19:02:18] ENGINE Serving on https://172.21.15.119:7150 2024-09-10T19:02:19.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:19 smithi119 ceph-mon[29577]: [10/Sep/2024:19:02:18] ENGINE Bus STARTED 2024-09-10T19:02:19.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:19 smithi119 ceph-mon[29577]: [10/Sep/2024:19:02:18] ENGINE Error in HTTPServer.serve 2024-09-10T19:02:19.319 INFO:journalctl@ceph.mon.a.smithi119.stdout: Traceback (most recent call last): 2024-09-10T19:02:19.319 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T19:02:19.319 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._connections.run(self.expiration_interval) 2024-09-10T19:02:19.319 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T19:02:19.319 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._run(expiration_interval) 2024-09-10T19:02:19.319 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T19:02:19.319 INFO:journalctl@ceph.mon.a.smithi119.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T19:02:19.320 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T19:02:19.320 INFO:journalctl@ceph.mon.a.smithi119.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T19:02:19.320 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T19:02:19.320 INFO:journalctl@ceph.mon.a.smithi119.stdout: s = self.context.wrap_socket( 2024-09-10T19:02:19.320 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T19:02:19.320 INFO:journalctl@ceph.mon.a.smithi119.stdout: return self.sslsocket_class._create( 2024-09-10T19:02:19.320 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T19:02:19.320 INFO:journalctl@ceph.mon.a.smithi119.stdout: self.do_handshake() 2024-09-10T19:02:19.320 INFO:journalctl@ceph.mon.a.smithi119.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T19:02:19.320 INFO:journalctl@ceph.mon.a.smithi119.stdout: self._sslobj.do_handshake() 2024-09-10T19:02:19.320 INFO:journalctl@ceph.mon.a.smithi119.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T19:02:19.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:19 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:20.387 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:02:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:02:20.535 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:20 smithi119 ceph-mon[29577]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:20.535 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:20 smithi119 ceph-mon[29577]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-10T19:02:20.535 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:20 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:20.535 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:20 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:20.535 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:20 smithi119 ceph-mon[40674]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:20.535 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:20 smithi119 ceph-mon[40674]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-10T19:02:20.535 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:20 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:20.535 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:20 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:20.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:20 smithi194 ceph-mon[32464]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:20.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:20 smithi194 ceph-mon[32464]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-10T19:02:20.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:20 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:20.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:20 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:20.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.762+0000 7ff963a8e640 1 -- 172.21.15.119:0/283403948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff95c106e70 msgr2=0x7ff95c109250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:20.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.762+0000 7ff963a8e640 1 --2- 172.21.15.119:0/283403948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff95c106e70 0x7ff95c109250 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff954009d20 tx=0x7ff954004c50 comp rx=0 tx=0).stop 2024-09-10T19:02:20.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.762+0000 7ff963a8e640 1 -- 172.21.15.119:0/283403948 shutdown_connections 2024-09-10T19:02:20.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.762+0000 7ff963a8e640 1 --2- 172.21.15.119:0/283403948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff95c106e70 0x7ff95c109250 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:20.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.762+0000 7ff963a8e640 1 --2- 172.21.15.119:0/283403948 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff95c0fe260 0x7ff95c106930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:20.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.762+0000 7ff963a8e640 1 --2- 172.21.15.119:0/283403948 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff95c0fd8c0 0x7ff95c0fdc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:20.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.762+0000 7ff963a8e640 1 -- 172.21.15.119:0/283403948 >> 172.21.15.119:0/283403948 conn(0x7ff95c071c00 msgr2=0x7ff95c072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:20.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.762+0000 7ff963a8e640 1 -- 172.21.15.119:0/283403948 shutdown_connections 2024-09-10T19:02:20.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.762+0000 7ff963a8e640 1 -- 172.21.15.119:0/283403948 wait complete. 2024-09-10T19:02:20.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.762+0000 7ff963a8e640 1 Processor -- start 2024-09-10T19:02:20.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.762+0000 7ff963a8e640 1 -- start start 2024-09-10T19:02:20.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff963a8e640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff95c0fd8c0 0x7ff95c112b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:20.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff963a8e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff95c0fe260 0x7ff95c113070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:20.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff963a8e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff95c106e70 0x7ff95c10ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:20.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff963a8e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c10d310 con 0x7ff95c0fe260 2024-09-10T19:02:20.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff963a8e640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c10d480 con 0x7ff95c0fd8c0 2024-09-10T19:02:20.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff963a8e640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c10d640 con 0x7ff95c106e70 2024-09-10T19:02:20.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff961029640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff95c0fe260 0x7ff95c113070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:20.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff961029640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff95c0fe260 0x7ff95c113070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51158/0 (socket says 172.21.15.119:51158) 2024-09-10T19:02:20.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff961029640 1 -- 172.21.15.119:0/4112205333 learned_addr learned my addr 172.21.15.119:0/4112205333 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:20.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff96202b640 1 --2- 172.21.15.119:0/4112205333 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff95c106e70 0x7ff95c10ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:20.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff96182a640 1 --2- 172.21.15.119:0/4112205333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff95c0fd8c0 0x7ff95c112b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:20.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff96202b640 1 -- 172.21.15.119:0/4112205333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff95c0fd8c0 msgr2=0x7ff95c112b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:20.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff96202b640 1 --2- 172.21.15.119:0/4112205333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff95c0fd8c0 0x7ff95c112b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:20.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff96202b640 1 -- 172.21.15.119:0/4112205333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff95c0fe260 msgr2=0x7ff95c113070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:20.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff96202b640 1 --2- 172.21.15.119:0/4112205333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff95c0fe260 0x7ff95c113070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:20.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff96202b640 1 -- 172.21.15.119:0/4112205333 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c005040 con 0x7ff95c106e70 2024-09-10T19:02:20.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff961029640 1 --2- 172.21.15.119:0/4112205333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff95c0fe260 0x7ff95c113070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:02:20.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff96202b640 1 --2- 172.21.15.119:0/4112205333 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff95c106e70 0x7ff95c10ccc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff954004ec0 tx=0x7ff954004c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:20.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff9527fc640 1 -- 172.21.15.119:0/4112205333 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95403c070 con 0x7ff95c106e70 2024-09-10T19:02:20.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.763+0000 7ff9527fc640 1 -- 172.21.15.119:0/4112205333 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7ff954040880 con 0x7ff95c106e70 2024-09-10T19:02:20.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.764+0000 7ff963a8e640 1 -- 172.21.15.119:0/4112205333 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9540098b0 con 0x7ff95c106e70 2024-09-10T19:02:20.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.764+0000 7ff9527fc640 1 -- 172.21.15.119:0/4112205333 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95403f6c0 con 0x7ff95c106e70 2024-09-10T19:02:20.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.764+0000 7ff963a8e640 1 -- 172.21.15.119:0/4112205333 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95c0fb6b0 con 0x7ff95c106e70 2024-09-10T19:02:20.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.765+0000 7ff963a8e640 1 -- 172.21.15.119:0/4112205333 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff95c065a80 con 0x7ff95c106e70 2024-09-10T19:02:20.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.765+0000 7ff9527fc640 1 -- 172.21.15.119:0/4112205333 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 21) v1 ==== 90748+0+0 (secure 0 0 0) 0x7ff954030070 con 0x7ff95c106e70 2024-09-10T19:02:20.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.766+0000 7ff9527fc640 1 --2- 172.21.15.119:0/4112205333 >> v2:172.21.15.119:6800/3118961733 conn(0x7ff94806ee40 0x7ff9480712f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:20.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.766+0000 7ff9527fc640 1 -- 172.21.15.119:0/4112205333 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7ff9540b3260 con 0x7ff95c106e70 2024-09-10T19:02:20.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.766+0000 7ff96182a640 1 --2- 172.21.15.119:0/4112205333 >> v2:172.21.15.119:6800/3118961733 conn(0x7ff94806ee40 0x7ff9480712f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:20.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.766+0000 7ff96182a640 1 --2- 172.21.15.119:0/4112205333 >> v2:172.21.15.119:6800/3118961733 conn(0x7ff94806ee40 0x7ff9480712f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff958009fd0 tx=0x7ff95800b040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:20.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.770+0000 7ff9527fc640 1 -- 172.21.15.119:0/4112205333 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7ff95407a2a0 con 0x7ff95c106e70 2024-09-10T19:02:20.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.940+0000 7ff963a8e640 1 -- 172.21.15.119:0/4112205333 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff95c10bd50 con 0x7ff95c106e70 2024-09-10T19:02:20.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.941+0000 7ff9527fc640 1 -- 172.21.15.119:0/4112205333 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+19257 (secure 0 0 0) 0x7ff954080250 con 0x7ff95c106e70 2024-09-10T19:02:20.943 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T19:02:20.943 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":61,"fsid":"3afe9804-6fa6-11ef-bcea-c7b262605968","created":"2024-09-10T18:56:26.299359+0000","modified":"2024-09-10T19:02:17.010185+0000","last_up_change":"2024-09-10T19:01:24.140998+0000","last_in_change":"2024-09-10T19:01:07.206749+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T18:59:30.453601+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-10T19:01:39.009038+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-10T19:01:39.241355+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":56,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-10T19:01:40.613730+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-10T19:01:42.567074+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"57","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-10T19:01:44.643080+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"7c2033fa-42d6-44b1-ac93-8e4ea2aea00f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6801","nonce":1598455131}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":1598455131}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":1598455131}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6803","nonce":1598455131}]},"public_addr":"172.21.15.119:6801/1598455131","cluster_addr":"172.21.15.119:6802/1598455131","heartbeat_back_addr":"172.21.15.119:6804/1598455131","heartbeat_front_addr":"172.21.15.119:6803/1598455131","state":["exists","up"]},{"osd":1,"uuid":"cdc97929-345b-440d-9d0f-c64e3cbe8110","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6805","nonce":4018071000}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":4018071000}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":4018071000}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6807","nonce":4018071000}]},"public_addr":"172.21.15.119:6805/4018071000","cluster_addr":"172.21.15.119:6806/4018071000","heartbeat_back_addr":"172.21.15.119:6808/4018071000","heartbeat_front_addr":"172.21.15.119:6807/4018071000","state":["exists","up"]},{"osd":2,"uuid":"a53d3e5b-f03d-4082-a6d5-af183c1e8ca4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6809","nonce":150641920}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":150641920}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":150641920}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6811","nonce":150641920}]},"public_addr":"172.21.15.119:6809/150641920","cluster_addr":"172.21.15.119:6810/150641920","heartbeat_back_addr":"172.21.15.119:6812/150641920","heartbeat_front_addr":"172.21.15.119:6811/150641920","state":["exists","up"]},{"osd":3,"uuid":"291e81da-6fc3-44a4-880c-dd6c0ad00c13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6813","nonce":783850966}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":783850966}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":783850966}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6815","nonce":783850966}]},"public_addr":"172.21.15.119:6813/783850966","cluster_addr":"172.21.15.119:6814/783850966","heartbeat_back_addr":"172.21.15.119:6816/783850966","heartbeat_front_addr":"172.21.15.119:6815/783850966","state":["exists","up"]},{"osd":4,"uuid":"0bee9361-6ab4-4545-a6d4-b05bbb4f9c51","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":4259870591}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6801","nonce":4259870591}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6803","nonce":4259870591}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":4259870591}]},"public_addr":"172.21.15.194:6800/4259870591","cluster_addr":"172.21.15.194:6801/4259870591","heartbeat_back_addr":"172.21.15.194:6803/4259870591","heartbeat_front_addr":"172.21.15.194:6802/4259870591","state":["exists","up"]},{"osd":5,"uuid":"b26af90f-5d90-4929-94e6-92545dffd78d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":635822671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6805","nonce":635822671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6807","nonce":635822671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":635822671}]},"public_addr":"172.21.15.194:6804/635822671","cluster_addr":"172.21.15.194:6805/635822671","heartbeat_back_addr":"172.21.15.194:6807/635822671","heartbeat_front_addr":"172.21.15.194:6806/635822671","state":["exists","up"]},{"osd":6,"uuid":"c05bb66f-8bc0-44b3-965d-fda246401108","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":2830881352}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6809","nonce":2830881352}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6811","nonce":2830881352}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":2830881352}]},"public_addr":"172.21.15.194:6808/2830881352","cluster_addr":"172.21.15.194:6809/2830881352","heartbeat_back_addr":"172.21.15.194:6811/2830881352","heartbeat_front_addr":"172.21.15.194:6810/2830881352","state":["exists","up"]},{"osd":7,"uuid":"16049fee-bcde-450e-b4c5-de13dbb392d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":117762113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6813","nonce":117762113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6815","nonce":117762113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":117762113}]},"public_addr":"172.21.15.194:6812/117762113","cluster_addr":"172.21.15.194:6813/117762113","heartbeat_back_addr":"172.21.15.194:6815/117762113","heartbeat_front_addr":"172.21.15.194:6814/117762113","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:58:39.672796+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:59:03.878834+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:59:27.525213+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T18:59:51.755950+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:00:12.576692+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:00:35.773984+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:00:58.811225+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:01:22.187718+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:0/284317922":"2024-09-11T19:02:17.010143+0000","172.21.15.119:6800/4184327284":"2024-09-11T18:56:55.126452+0000","172.21.15.119:0/4285164139":"2024-09-11T18:56:55.126452+0000","172.21.15.119:0/3881701229":"2024-09-11T18:56:55.126452+0000","172.21.15.119:0/313114785":"2024-09-11T19:02:17.010143+0000","172.21.15.119:0/359148006":"2024-09-11T18:56:55.126452+0000","172.21.15.119:0/1889495967":"2024-09-11T18:57:18.128849+0000","172.21.15.119:6800/1136125279":"2024-09-11T18:57:18.128849+0000","172.21.15.119:6800/1094129222":"2024-09-11T19:02:17.010143+0000","172.21.15.119:0/651881751":"2024-09-11T18:57:18.128849+0000","172.21.15.119:0/667233824":"2024-09-11T18:57:18.128849+0000","172.21.15.119:0/3164939798":"2024-09-11T19:02:17.010143+0000","172.21.15.119:0/624043880":"2024-09-11T19:02:17.010143+0000","172.21.15.119:0/1352141759":"2024-09-11T19:02:17.010143+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:02:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.943+0000 7ff963a8e640 1 -- 172.21.15.119:0/4112205333 >> v2:172.21.15.119:6800/3118961733 conn(0x7ff94806ee40 msgr2=0x7ff9480712f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.943+0000 7ff963a8e640 1 --2- 172.21.15.119:0/4112205333 >> v2:172.21.15.119:6800/3118961733 conn(0x7ff94806ee40 0x7ff9480712f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff958009fd0 tx=0x7ff95800b040 comp rx=0 tx=0).stop 2024-09-10T19:02:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.943+0000 7ff963a8e640 1 -- 172.21.15.119:0/4112205333 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff95c106e70 msgr2=0x7ff95c10ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.943+0000 7ff963a8e640 1 --2- 172.21.15.119:0/4112205333 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff95c106e70 0x7ff95c10ccc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff954004ec0 tx=0x7ff954004c50 comp rx=0 tx=0).stop 2024-09-10T19:02:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.943+0000 7ff963a8e640 1 -- 172.21.15.119:0/4112205333 shutdown_connections 2024-09-10T19:02:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.943+0000 7ff963a8e640 1 --2- 172.21.15.119:0/4112205333 >> v2:172.21.15.119:6800/3118961733 conn(0x7ff94806ee40 0x7ff9480712f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.943+0000 7ff963a8e640 1 --2- 172.21.15.119:0/4112205333 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff95c106e70 0x7ff95c10ccc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.943+0000 7ff963a8e640 1 --2- 172.21.15.119:0/4112205333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff95c0fe260 0x7ff95c113070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.943+0000 7ff963a8e640 1 --2- 172.21.15.119:0/4112205333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff95c0fd8c0 0x7ff95c112b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.943+0000 7ff963a8e640 1 -- 172.21.15.119:0/4112205333 >> 172.21.15.119:0/4112205333 conn(0x7ff95c071c00 msgr2=0x7ff95c0fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.943+0000 7ff963a8e640 1 -- 172.21.15.119:0/4112205333 shutdown_connections 2024-09-10T19:02:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:20.943+0000 7ff963a8e640 1 -- 172.21.15.119:0/4112205333 wait complete. 2024-09-10T19:02:21.789 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-10T19:02:21.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-10T19:02:21.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-10T19:02:21.790 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-10T19:02:21.791 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-10T19:02:21.791 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-10T19:02:21.792 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-10T19:02:21.792 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-10T19:02:22.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:21 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:21 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:21 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:22.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:21 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:22.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:21 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:22.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:21 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:22.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:21 smithi194 ceph-mon[32464]: Adjusting osd_memory_target on smithi194 to 3760M 2024-09-10T19:02:22.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:21 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:21 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:21 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4112205333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:02:22.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:21 smithi194 ceph-mon[32464]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:22.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:22.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:22.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:22.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:22.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[40674]: Adjusting osd_memory_target on smithi194 to 3760M 2024-09-10T19:02:22.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4112205333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:02:22.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[40674]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:22.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:22.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:22.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:22.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:22.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[29577]: Adjusting osd_memory_target on smithi194 to 3760M 2024-09-10T19:02:22.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:22.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4112205333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:02:22.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:21 smithi119 ceph-mon[29577]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:22.266 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:22.325 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:22.328 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:22.329 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:22.333 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:22.336 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:23.016 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:23.016 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:23.016 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:23.016 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:23.016 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:23.016 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:23.017 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: Adjusting osd_memory_target on smithi119 to 3760M 2024-09-10T19:02:23.017 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:23.017 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:02:23.017 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:02:23.017 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-10T19:02:23.017 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-10T19:02:23.017 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-10T19:02:23.017 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:02:23.017 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:02:23.017 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: Updating smithi119:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:02:23.017 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T19:02:23.018 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:02:23.018 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:23.018 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:23.018 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:23.018 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:23.018 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:23.018 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:23.018 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:23.018 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:23.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: Adjusting osd_memory_target on smithi119 to 3760M 2024-09-10T19:02:23.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:23.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:02:23.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:02:23.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-10T19:02:23.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-10T19:02:23.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-10T19:02:23.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:02:23.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:02:23.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: Updating smithi119:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:02:23.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T19:02:23.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:02:23.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:23.019 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:23.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:23.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:23.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:23.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:23.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:23.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:02:23.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: Adjusting osd_memory_target on smithi119 to 3760M 2024-09-10T19:02:23.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:23.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:02:23.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:02:23.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: Updating smithi119:/etc/ceph/ceph.conf 2024-09-10T19:02:23.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: Updating smithi194:/etc/ceph/ceph.conf 2024-09-10T19:02:23.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-10T19:02:23.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:02:23.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:02:23.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: Updating smithi119:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:02:23.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: Updating smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T19:02:23.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: Updating smithi119:/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:02:23.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:23.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:24.055 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:23 smithi119 ceph-mon[40674]: Updating smithi119:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T19:02:24.056 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:23 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:24.056 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:23 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:24.056 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:23 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:24.056 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:23 smithi119 ceph-mon[40674]: Deploying daemon alertmanager.a on smithi119 2024-09-10T19:02:24.056 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:23 smithi119 ceph-mon[40674]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:24.056 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:23 smithi119 ceph-mon[29577]: Updating smithi119:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T19:02:24.056 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:23 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:24.056 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:23 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:24.056 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:23 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:24.056 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:23 smithi119 ceph-mon[29577]: Deploying daemon alertmanager.a on smithi119 2024-09-10T19:02:24.057 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:23 smithi119 ceph-mon[29577]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:24.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:23 smithi194 ceph-mon[32464]: Updating smithi119:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T19:02:24.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:23 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:24.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:23 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:24.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:23 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:24.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:23 smithi194 ceph-mon[32464]: Deploying daemon alertmanager.a on smithi119 2024-09-10T19:02:24.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:23 smithi194 ceph-mon[32464]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:02:26.298 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:26.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:26 smithi119 ceph-mon[29577]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-10T19:02:26.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:26 smithi119 ceph-mon[40674]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-10T19:02:26.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:26 smithi194 ceph-mon[32464]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-10T19:02:28.216 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:28 smithi119 ceph-mon[29577]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T19:02:28.216 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:28 smithi119 ceph-mon[40674]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T19:02:28.320 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:28 smithi194 ceph-mon[32464]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T19:02:28.321 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:02:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:02:29.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.977+0000 7fc18a714640 1 -- 172.21.15.119:0/2684891211 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1840fc1b0 msgr2=0x7fc184108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:29.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.977+0000 7fc18a714640 1 --2- 172.21.15.119:0/2684891211 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1840fc1b0 0x7fc184108580 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009d00 tx=0x7fc17c004d10 comp rx=0 tx=0).stop 2024-09-10T19:02:29.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.977+0000 7fc18a714640 1 -- 172.21.15.119:0/2684891211 shutdown_connections 2024-09-10T19:02:29.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.977+0000 7fc18a714640 1 --2- 172.21.15.119:0/2684891211 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1840fc1b0 0x7fc184108580 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:29.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.977+0000 7fc18a714640 1 --2- 172.21.15.119:0/2684891211 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc1840fb800 0x7fc1840fbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:29.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.977+0000 7fc18a714640 1 --2- 172.21.15.119:0/2684891211 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1841017d0 0x7fc184101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:29.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.977+0000 7fc18a714640 1 -- 172.21.15.119:0/2684891211 >> 172.21.15.119:0/2684891211 conn(0x7fc184071c00 msgr2=0x7fc184072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:29.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.977+0000 7fc18a714640 1 -- 172.21.15.119:0/2684891211 shutdown_connections 2024-09-10T19:02:29.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.977+0000 7fc18a714640 1 -- 172.21.15.119:0/2684891211 wait complete. 2024-09-10T19:02:29.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.978+0000 7fc18a714640 1 Processor -- start 2024-09-10T19:02:29.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.978+0000 7fc18a714640 1 -- start start 2024-09-10T19:02:29.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.978+0000 7fc18a714640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1840fb800 0x7fc184111a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:29.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.978+0000 7fc18a714640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc1840fc1b0 0x7fc184111fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:29.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.978+0000 7fc18a714640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1841017d0 0x7fc18410bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:29.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.978+0000 7fc18a714640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18410c200 con 0x7fc1841017d0 2024-09-10T19:02:29.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.978+0000 7fc18a714640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18410c370 con 0x7fc1840fc1b0 2024-09-10T19:02:29.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.978+0000 7fc18a714640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc18410c530 con 0x7fc1840fb800 2024-09-10T19:02:29.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc188cb1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1841017d0 0x7fc18410bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:29.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc183fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1840fb800 0x7fc184111a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:29.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc188cb1640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1841017d0 0x7fc18410bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49284/0 (socket says 172.21.15.119:49284) 2024-09-10T19:02:29.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc188cb1640 1 -- 172.21.15.119:0/4225804220 learned_addr learned my addr 172.21.15.119:0/4225804220 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:29.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc1837fe640 1 --2- 172.21.15.119:0/4225804220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc1840fc1b0 0x7fc184111fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:29.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc188cb1640 1 -- 172.21.15.119:0/4225804220 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1840fb800 msgr2=0x7fc184111a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:29.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc188cb1640 1 --2- 172.21.15.119:0/4225804220 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1840fb800 0x7fc184111a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:29.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc188cb1640 1 -- 172.21.15.119:0/4225804220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc1840fc1b0 msgr2=0x7fc184111fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:29.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc188cb1640 1 --2- 172.21.15.119:0/4225804220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc1840fc1b0 0x7fc184111fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:29.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc188cb1640 1 -- 172.21.15.119:0/4225804220 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc174005040 con 0x7fc1841017d0 2024-09-10T19:02:29.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc188cb1640 1 --2- 172.21.15.119:0/4225804220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1841017d0 0x7fc18410bc30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc17c02a840 tx=0x7fc17c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:29.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc180ff9640 1 -- 172.21.15.119:0/4225804220 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c038070 con 0x7fc1841017d0 2024-09-10T19:02:29.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc180ff9640 1 -- 172.21.15.119:0/4225804220 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fc17c033e10 con 0x7fc1841017d0 2024-09-10T19:02:29.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc18a714640 1 -- 172.21.15.119:0/4225804220 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c0098b0 con 0x7fc1841017d0 2024-09-10T19:02:29.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc180ff9640 1 -- 172.21.15.119:0/4225804220 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc17c03ca50 con 0x7fc1841017d0 2024-09-10T19:02:29.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.979+0000 7fc18a714640 1 -- 172.21.15.119:0/4225804220 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1841188f0 con 0x7fc1841017d0 2024-09-10T19:02:29.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.980+0000 7fc18a714640 1 -- 172.21.15.119:0/4225804220 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc1840fa690 con 0x7fc1841017d0 2024-09-10T19:02:29.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.981+0000 7fc180ff9640 1 -- 172.21.15.119:0/4225804220 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7fc17c030070 con 0x7fc1841017d0 2024-09-10T19:02:29.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.981+0000 7fc180ff9640 1 --2- 172.21.15.119:0/4225804220 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc16c06eed0 0x7fc16c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:29.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.982+0000 7fc183fff640 1 --2- 172.21.15.119:0/4225804220 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc16c06eed0 0x7fc16c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:29.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.982+0000 7fc180ff9640 1 -- 172.21.15.119:0/4225804220 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7fc17c0aedf0 con 0x7fc1841017d0 2024-09-10T19:02:29.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.982+0000 7fc180ff9640 1 --2- 172.21.15.119:0/4225804220 >> v2:172.21.15.119:6805/4018071000 conn(0x7fc16c078800 0x7fc16c07ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:29.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.982+0000 7fc180ff9640 1 -- 172.21.15.119:0/4225804220 --> v2:172.21.15.119:6805/4018071000 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc16c07b2c0 con 0x7fc16c078800 2024-09-10T19:02:29.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.982+0000 7fc180ff9640 1 -- 172.21.15.119:0/4225804220 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc17c0b1430 con 0x7fc1841017d0 2024-09-10T19:02:29.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.982+0000 7fc1837fe640 1 --2- 172.21.15.119:0/4225804220 >> v2:172.21.15.119:6805/4018071000 conn(0x7fc16c078800 0x7fc16c07ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:29.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.982+0000 7fc183fff640 1 --2- 172.21.15.119:0/4225804220 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc16c06eed0 0x7fc16c071380 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc174004a50 tx=0x7fc17400b040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:29.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.982+0000 7fc1837fe640 1 --2- 172.21.15.119:0/4225804220 >> v2:172.21.15.119:6805/4018071000 conn(0x7fc16c078800 0x7fc16c07ac30 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:29.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.983+0000 7fc180ff9640 1 -- 172.21.15.119:0/4225804220 <== osd.1 v2:172.21.15.119:6805/4018071000 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fc16c07b2c0 con 0x7fc16c078800 2024-09-10T19:02:29.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.996+0000 7fc18a714640 1 -- 172.21.15.119:0/4225804220 --> v2:172.21.15.119:6805/4018071000 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc1840fcef0 con 0x7fc16c078800 2024-09-10T19:02:29.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.997+0000 7fc180ff9640 1 -- 172.21.15.119:0/4225804220 <== osd.1 v2:172.21.15.119:6805/4018071000 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc1840fcef0 con 0x7fc16c078800 2024-09-10T19:02:29.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.997+0000 7fc18a714640 1 -- 172.21.15.119:0/4225804220 >> v2:172.21.15.119:6805/4018071000 conn(0x7fc16c078800 msgr2=0x7fc16c07ac30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:29.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.997+0000 7fc18a714640 1 --2- 172.21.15.119:0/4225804220 >> v2:172.21.15.119:6805/4018071000 conn(0x7fc16c078800 0x7fc16c07ac30 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:29.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.997+0000 7fc18a714640 1 -- 172.21.15.119:0/4225804220 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc16c06eed0 msgr2=0x7fc16c071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:29.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.997+0000 7fc18a714640 1 --2- 172.21.15.119:0/4225804220 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc16c06eed0 0x7fc16c071380 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fc174004a50 tx=0x7fc17400b040 comp rx=0 tx=0).stop 2024-09-10T19:02:29.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.997+0000 7fc18a714640 1 -- 172.21.15.119:0/4225804220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1841017d0 msgr2=0x7fc18410bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:29.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.997+0000 7fc18a714640 1 --2- 172.21.15.119:0/4225804220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1841017d0 0x7fc18410bc30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fc17c02a840 tx=0x7fc17c004d10 comp rx=0 tx=0).stop 2024-09-10T19:02:29.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.997+0000 7fc188cb1640 1 -- 172.21.15.119:0/4225804220 reap_dead start 2024-09-10T19:02:29.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.997+0000 7fc18a714640 1 -- 172.21.15.119:0/4225804220 shutdown_connections 2024-09-10T19:02:29.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.997+0000 7fc18a714640 1 -- 172.21.15.119:0/4225804220 >> 172.21.15.119:0/4225804220 conn(0x7fc184071c00 msgr2=0x7fc1840f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:29.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.997+0000 7fc18a714640 1 -- 172.21.15.119:0/4225804220 shutdown_connections 2024-09-10T19:02:29.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:28.997+0000 7fc18a714640 1 -- 172.21.15.119:0/4225804220 wait complete. 2024-09-10T19:02:29.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:29 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:29 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:29.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:29 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:30.500 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:30 smithi119 ceph-mon[29577]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T19:02:30.501 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:30 smithi119 ceph-mon[40674]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T19:02:30.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:30 smithi194 ceph-mon[32464]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T19:02:30.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:02:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:02:32.272 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:32 smithi119 ceph-mon[29577]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-10T19:02:32.272 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:32.272 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:32 smithi119 ceph-mon[40674]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-10T19:02:32.272 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:32.277 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:32.300 INFO:teuthology.orchestra.run.smithi119.stdout:55834574892 2024-09-10T19:02:32.300 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-10T19:02:32.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:32 smithi194 ceph-mon[32464]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 6 op/s 2024-09-10T19:02:32.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:33.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 -- 172.21.15.119:0/2416150353 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f31e00fca50 msgr2=0x7f31e00fcec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:33.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 --2- 172.21.15.119:0/2416150353 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f31e00fca50 0x7f31e00fcec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f31d4004980 tx=0x7f31d4009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 -- 172.21.15.119:0/2416150353 shutdown_connections 2024-09-10T19:02:33.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 --2- 172.21.15.119:0/2416150353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31e00fd400 0x7f31e0109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 --2- 172.21.15.119:0/2416150353 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f31e00fca50 0x7f31e00fcec0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 --2- 172.21.15.119:0/2416150353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31e01028e0 0x7f31e0102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 -- 172.21.15.119:0/2416150353 >> 172.21.15.119:0/2416150353 conn(0x7f31e0073600 msgr2=0x7f31e0073a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:33.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 -- 172.21.15.119:0/2416150353 shutdown_connections 2024-09-10T19:02:33.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 -- 172.21.15.119:0/2416150353 wait complete. 2024-09-10T19:02:33.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 Processor -- start 2024-09-10T19:02:33.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 -- start start 2024-09-10T19:02:33.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f31e00fca50 0x7f31e0112dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:33.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31e00fd400 0x7f31e0113310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:33.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31e01028e0 0x7f31e010cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:33.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e010d5b0 con 0x7f31e00fd400 2024-09-10T19:02:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e010d720 con 0x7f31e01028e0 2024-09-10T19:02:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.066+0000 7f31e83db640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e010d8e0 con 0x7f31e00fca50 2024-09-10T19:02:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31e6177640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f31e00fca50 0x7f31e0112dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31e6177640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f31e00fca50 0x7f31e0112dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:58772/0 (socket says 172.21.15.119:58772) 2024-09-10T19:02:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31e6177640 1 -- 172.21.15.119:0/846441524 learned_addr learned my addr 172.21.15.119:0/846441524 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31e5976640 1 --2- 172.21.15.119:0/846441524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31e00fd400 0x7f31e0113310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31e6978640 1 --2- 172.21.15.119:0/846441524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31e01028e0 0x7f31e010cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31e6177640 1 -- 172.21.15.119:0/846441524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31e01028e0 msgr2=0x7f31e010cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31e6177640 1 --2- 172.21.15.119:0/846441524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31e01028e0 0x7f31e010cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31e6177640 1 -- 172.21.15.119:0/846441524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31e00fd400 msgr2=0x7f31e0113310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31e6177640 1 --2- 172.21.15.119:0/846441524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31e00fd400 0x7f31e0113310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31e6177640 1 -- 172.21.15.119:0/846441524 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d00098b0 con 0x7f31e00fca50 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31e5976640 1 --2- 172.21.15.119:0/846441524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31e00fd400 0x7f31e0113310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31e6177640 1 --2- 172.21.15.119:0/846441524 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f31e00fca50 0x7f31e0112dd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f31d0001f00 tx=0x7f31d0004c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31deffd640 1 -- 172.21.15.119:0/846441524 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d000fea0 con 0x7f31e00fca50 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31deffd640 1 -- 172.21.15.119:0/846441524 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f31d0007d00 con 0x7f31e00fca50 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31e83db640 1 -- 172.21.15.119:0/846441524 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d4005040 con 0x7f31e00fca50 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31deffd640 1 -- 172.21.15.119:0/846441524 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d0015470 con 0x7f31e00fca50 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.067+0000 7f31e83db640 1 -- 172.21.15.119:0/846441524 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e0119ca0 con 0x7f31e00fca50 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.068+0000 7f31e83db640 1 -- 172.21.15.119:0/846441524 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f31e005dd90 con 0x7f31e00fca50 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.069+0000 7f31deffd640 1 -- 172.21.15.119:0/846441524 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f31d0007890 con 0x7f31e00fca50 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.069+0000 7f31deffd640 1 --2- 172.21.15.119:0/846441524 >> v2:172.21.15.119:6800/3118961733 conn(0x7f31c806eed0 0x7f31c8071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.069+0000 7f31e5976640 1 --2- 172.21.15.119:0/846441524 >> v2:172.21.15.119:6800/3118961733 conn(0x7f31c806eed0 0x7f31c8071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.069+0000 7f31deffd640 1 -- 172.21.15.119:0/846441524 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f31d008e8a0 con 0x7f31e00fca50 2024-09-10T19:02:33.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.069+0000 7f31deffd640 1 --2- 172.21.15.119:0/846441524 >> v2:172.21.15.119:6809/150641920 conn(0x7f31c8078800 0x7f31c807ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:33.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.069+0000 7f31deffd640 1 -- 172.21.15.119:0/846441524 --> v2:172.21.15.119:6809/150641920 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f31c807b2c0 con 0x7f31c8078800 2024-09-10T19:02:33.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.070+0000 7f31deffd640 1 -- 172.21.15.119:0/846441524 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31d0093240 con 0x7f31e00fca50 2024-09-10T19:02:33.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.070+0000 7f31e6978640 1 --2- 172.21.15.119:0/846441524 >> v2:172.21.15.119:6809/150641920 conn(0x7f31c8078800 0x7f31c807ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:33.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.070+0000 7f31e5976640 1 --2- 172.21.15.119:0/846441524 >> v2:172.21.15.119:6800/3118961733 conn(0x7f31c806eed0 0x7f31c8071380 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009f80 tx=0x7f31d4009da0 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:33.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.070+0000 7f31e6978640 1 --2- 172.21.15.119:0/846441524 >> v2:172.21.15.119:6809/150641920 conn(0x7f31c8078800 0x7f31c807ac30 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:33.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.071+0000 7f31deffd640 1 -- 172.21.15.119:0/846441524 <== osd.2 v2:172.21.15.119:6809/150641920 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f31c807b2c0 con 0x7f31c8078800 2024-09-10T19:02:33.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.083+0000 7f31e83db640 1 -- 172.21.15.119:0/846441524 --> v2:172.21.15.119:6809/150641920 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f31e00fe000 con 0x7f31c8078800 2024-09-10T19:02:33.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.083+0000 7f31deffd640 1 -- 172.21.15.119:0/846441524 <== osd.2 v2:172.21.15.119:6809/150641920 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f31e00fe000 con 0x7f31c8078800 2024-09-10T19:02:33.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.083+0000 7f31e83db640 1 -- 172.21.15.119:0/846441524 >> v2:172.21.15.119:6809/150641920 conn(0x7f31c8078800 msgr2=0x7f31c807ac30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:33.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.083+0000 7f31e83db640 1 --2- 172.21.15.119:0/846441524 >> v2:172.21.15.119:6809/150641920 conn(0x7f31c8078800 0x7f31c807ac30 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.083+0000 7f31e83db640 1 -- 172.21.15.119:0/846441524 >> v2:172.21.15.119:6800/3118961733 conn(0x7f31c806eed0 msgr2=0x7f31c8071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:33.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.083+0000 7f31e83db640 1 --2- 172.21.15.119:0/846441524 >> v2:172.21.15.119:6800/3118961733 conn(0x7f31c806eed0 0x7f31c8071380 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f31d4009f80 tx=0x7f31d4009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.083+0000 7f31e83db640 1 -- 172.21.15.119:0/846441524 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f31e00fca50 msgr2=0x7f31e0112dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:33.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.083+0000 7f31e83db640 1 --2- 172.21.15.119:0/846441524 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f31e00fca50 0x7f31e0112dd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f31d0001f00 tx=0x7f31d0004c50 comp rx=0 tx=0).stop 2024-09-10T19:02:33.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.083+0000 7f31e6978640 1 -- 172.21.15.119:0/846441524 reap_dead start 2024-09-10T19:02:33.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.083+0000 7f31e83db640 1 -- 172.21.15.119:0/846441524 shutdown_connections 2024-09-10T19:02:33.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.083+0000 7f31e83db640 1 -- 172.21.15.119:0/846441524 >> 172.21.15.119:0/846441524 conn(0x7f31e0073600 msgr2=0x7f31e00fb710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:33.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.083+0000 7f31e83db640 1 -- 172.21.15.119:0/846441524 shutdown_connections 2024-09-10T19:02:33.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.083+0000 7f31e83db640 1 -- 172.21.15.119:0/846441524 wait complete. 2024-09-10T19:02:33.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.280+0000 7f5aafd3a640 1 -- 172.21.15.119:0/1273871805 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aa80fc8a0 msgr2=0x7f5aa80fcd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:33.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.280+0000 7f5aafd3a640 1 --2- 172.21.15.119:0/1273871805 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aa80fc8a0 0x7f5aa80fcd10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c004980 tx=0x7f5a9c009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.280+0000 7f5aafd3a640 1 -- 172.21.15.119:0/1273871805 shutdown_connections 2024-09-10T19:02:33.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.280+0000 7f5aafd3a640 1 --2- 172.21.15.119:0/1273871805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5aa80fd250 0x7f5aa8109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.280+0000 7f5aafd3a640 1 --2- 172.21.15.119:0/1273871805 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aa80fc8a0 0x7f5aa80fcd10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.280+0000 7f5aafd3a640 1 --2- 172.21.15.119:0/1273871805 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aa81028c0 0x7f5aa8102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.280+0000 7f5aafd3a640 1 -- 172.21.15.119:0/1273871805 >> 172.21.15.119:0/1273871805 conn(0x7f5aa80f8400 msgr2=0x7f5aa80fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:33.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.280+0000 7f5aafd3a640 1 -- 172.21.15.119:0/1273871805 shutdown_connections 2024-09-10T19:02:33.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.280+0000 7f5aafd3a640 1 -- 172.21.15.119:0/1273871805 wait complete. 2024-09-10T19:02:33.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.280+0000 7f5aafd3a640 1 Processor -- start 2024-09-10T19:02:33.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.280+0000 7f5aafd3a640 1 -- start start 2024-09-10T19:02:33.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aafd3a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aa80fc8a0 0x7f5aa8112dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:33.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aafd3a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aa80fd250 0x7f5aa8113310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:33.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aafd3a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5aa81028c0 0x7f5aa810cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:33.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aafd3a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa810d5b0 con 0x7f5aa80fd250 2024-09-10T19:02:33.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aafd3a640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa810d720 con 0x7f5aa81028c0 2024-09-10T19:02:33.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aafd3a640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa810d8e0 con 0x7f5aa80fc8a0 2024-09-10T19:02:33.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aad2d5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aa80fd250 0x7f5aa8113310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:33.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aad2d5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aa80fd250 0x7f5aa8113310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36732/0 (socket says 172.21.15.119:36732) 2024-09-10T19:02:33.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aad2d5640 1 -- 172.21.15.119:0/254609116 learned_addr learned my addr 172.21.15.119:0/254609116 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:33.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aadad6640 1 --2- 172.21.15.119:0/254609116 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aa80fc8a0 0x7f5aa8112dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:33.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aae2d7640 1 --2- 172.21.15.119:0/254609116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5aa81028c0 0x7f5aa810cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:33.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aad2d5640 1 -- 172.21.15.119:0/254609116 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aa80fc8a0 msgr2=0x7f5aa8112dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:33.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aad2d5640 1 --2- 172.21.15.119:0/254609116 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5aa80fc8a0 0x7f5aa8112dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aad2d5640 1 -- 172.21.15.119:0/254609116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5aa81028c0 msgr2=0x7f5aa810cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:33.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aad2d5640 1 --2- 172.21.15.119:0/254609116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5aa81028c0 0x7f5aa810cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aad2d5640 1 -- 172.21.15.119:0/254609116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa00098b0 con 0x7f5aa80fd250 2024-09-10T19:02:33.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aad2d5640 1 --2- 172.21.15.119:0/254609116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aa80fd250 0x7f5aa8113310 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c0094b0 tx=0x7f5a9c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:33.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aa67fc640 1 -- 172.21.15.119:0/254609116 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c02bcb0 con 0x7f5aa80fd250 2024-09-10T19:02:33.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aafd3a640 1 -- 172.21.15.119:0/254609116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9c005040 con 0x7f5aa80fd250 2024-09-10T19:02:33.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aa67fc640 1 -- 172.21.15.119:0/254609116 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f5a9c02be10 con 0x7f5aa80fd250 2024-09-10T19:02:33.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aa67fc640 1 -- 172.21.15.119:0/254609116 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a9c0336b0 con 0x7f5aa80fd250 2024-09-10T19:02:33.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.281+0000 7f5aafd3a640 1 -- 172.21.15.119:0/254609116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa8119c20 con 0x7f5aa80fd250 2024-09-10T19:02:33.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.282+0000 7f5aafd3a640 1 -- 172.21.15.119:0/254609116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5aa805ddd0 con 0x7f5aa80fd250 2024-09-10T19:02:33.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.283+0000 7f5aa67fc640 1 -- 172.21.15.119:0/254609116 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f5a9c030020 con 0x7f5aa80fd250 2024-09-10T19:02:33.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.283+0000 7f5aa67fc640 1 --2- 172.21.15.119:0/254609116 >> v2:172.21.15.119:6800/3118961733 conn(0x7f5a9006eed0 0x7f5a90071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:33.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.283+0000 7f5aa67fc640 1 -- 172.21.15.119:0/254609116 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f5a9c0ae0c0 con 0x7f5aa80fd250 2024-09-10T19:02:33.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.283+0000 7f5aadad6640 1 --2- 172.21.15.119:0/254609116 >> v2:172.21.15.119:6800/3118961733 conn(0x7f5a9006eed0 0x7f5a90071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:33.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.283+0000 7f5aa67fc640 1 --2- 172.21.15.119:0/254609116 >> v2:172.21.15.194:6812/117762113 conn(0x7f5a90078780 0x7f5a9007abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:33.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.284+0000 7f5aa67fc640 1 -- 172.21.15.119:0/254609116 --> v2:172.21.15.194:6812/117762113 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a9007b240 con 0x7f5a90078780 2024-09-10T19:02:33.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.284+0000 7f5aa67fc640 1 -- 172.21.15.119:0/254609116 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a9c0b0700 con 0x7f5aa80fd250 2024-09-10T19:02:33.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.284+0000 7f5aadad6640 1 --2- 172.21.15.119:0/254609116 >> v2:172.21.15.119:6800/3118961733 conn(0x7f5a9006eed0 0x7f5a90071380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0001610 tx=0x7f5aa0004c50 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:33.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.284+0000 7f5aae2d7640 1 --2- 172.21.15.119:0/254609116 >> v2:172.21.15.194:6812/117762113 conn(0x7f5a90078780 0x7f5a9007abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:33.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.284+0000 7f5aae2d7640 1 --2- 172.21.15.119:0/254609116 >> v2:172.21.15.194:6812/117762113 conn(0x7f5a90078780 0x7f5a9007abb0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:33.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.285+0000 7f5aa67fc640 1 -- 172.21.15.119:0/254609116 <== osd.7 v2:172.21.15.194:6812/117762113 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f5a9007b240 con 0x7f5a90078780 2024-09-10T19:02:33.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.297+0000 7f5aafd3a640 1 -- 172.21.15.119:0/254609116 --> v2:172.21.15.194:6812/117762113 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5aa805e180 con 0x7f5a90078780 2024-09-10T19:02:33.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.298+0000 7f5aa67fc640 1 -- 172.21.15.119:0/254609116 <== osd.7 v2:172.21.15.194:6812/117762113 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5aa805e180 con 0x7f5a90078780 2024-09-10T19:02:33.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.298+0000 7f5aafd3a640 1 -- 172.21.15.119:0/254609116 >> v2:172.21.15.194:6812/117762113 conn(0x7f5a90078780 msgr2=0x7f5a9007abb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:33.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.298+0000 7f5aafd3a640 1 --2- 172.21.15.119:0/254609116 >> v2:172.21.15.194:6812/117762113 conn(0x7f5a90078780 0x7f5a9007abb0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.298+0000 7f5aafd3a640 1 -- 172.21.15.119:0/254609116 >> v2:172.21.15.119:6800/3118961733 conn(0x7f5a9006eed0 msgr2=0x7f5a90071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:33.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.298+0000 7f5aafd3a640 1 --2- 172.21.15.119:0/254609116 >> v2:172.21.15.119:6800/3118961733 conn(0x7f5a9006eed0 0x7f5a90071380 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5aa0001610 tx=0x7f5aa0004c50 comp rx=0 tx=0).stop 2024-09-10T19:02:33.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.298+0000 7f5aafd3a640 1 -- 172.21.15.119:0/254609116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aa80fd250 msgr2=0x7f5aa8113310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:33.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.298+0000 7f5aafd3a640 1 --2- 172.21.15.119:0/254609116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5aa80fd250 0x7f5aa8113310 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c0094b0 tx=0x7f5a9c009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:33.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.298+0000 7f5aae2d7640 1 -- 172.21.15.119:0/254609116 reap_dead start 2024-09-10T19:02:33.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.298+0000 7f5aafd3a640 1 -- 172.21.15.119:0/254609116 shutdown_connections 2024-09-10T19:02:33.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.298+0000 7f5aafd3a640 1 -- 172.21.15.119:0/254609116 >> 172.21.15.119:0/254609116 conn(0x7f5aa80f8400 msgr2=0x7f5aa80f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:33.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.298+0000 7f5aafd3a640 1 -- 172.21.15.119:0/254609116 shutdown_connections 2024-09-10T19:02:33.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:33.299+0000 7f5aafd3a640 1 -- 172.21.15.119:0/254609116 wait complete. 2024-09-10T19:02:33.961 INFO:teuthology.orchestra.run.smithi119.stdout:206158430224 2024-09-10T19:02:33.961 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-10T19:02:34.195 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:34 smithi119 ceph-mon[29577]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T19:02:34.195 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:34 smithi119 ceph-mon[40674]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T19:02:34.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:34 smithi194 ceph-mon[32464]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T19:02:35.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.976+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/3674686315 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbdc80fb950 msgr2=0x7fbdc80fbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:35.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.976+0000 7fbdcd4e5640 1 --2- 172.21.15.119:0/3674686315 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbdc80fb950 0x7fbdc80fbda0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc004950 tx=0x7fbdbc009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:35.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.976+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/3674686315 shutdown_connections 2024-09-10T19:02:35.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.977+0000 7fbdcd4e5640 1 --2- 172.21.15.119:0/3674686315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdc80fc2e0 0x7fbdc8108530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:35.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.977+0000 7fbdcd4e5640 1 --2- 172.21.15.119:0/3674686315 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbdc80fb950 0x7fbdc80fbda0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:35.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.977+0000 7fbdcd4e5640 1 --2- 172.21.15.119:0/3674686315 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdc81017e0 0x7fbdc8101bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:35.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.977+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/3674686315 >> 172.21.15.119:0/3674686315 conn(0x7fbdc8072540 msgr2=0x7fbdc8072940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:35.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.977+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/3674686315 shutdown_connections 2024-09-10T19:02:35.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.977+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/3674686315 wait complete. 2024-09-10T19:02:35.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.977+0000 7fbdcd4e5640 1 Processor -- start 2024-09-10T19:02:35.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.977+0000 7fbdcd4e5640 1 -- start start 2024-09-10T19:02:35.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.977+0000 7fbdcd4e5640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdc80fb950 0x7fbdc810df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:35.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.977+0000 7fbdcd4e5640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbdc80fc2e0 0x7fbdc810e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:35.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.977+0000 7fbdcd4e5640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdc81017e0 0x7fbdc810e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:35.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.977+0000 7fbdcd4e5640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc8119150 con 0x7fbdc81017e0 2024-09-10T19:02:35.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.977+0000 7fbdcd4e5640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc81192c0 con 0x7fbdc80fb950 2024-09-10T19:02:35.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.977+0000 7fbdcd4e5640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc8119480 con 0x7fbdc80fc2e0 2024-09-10T19:02:35.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdc67fc640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbdc80fc2e0 0x7fbdc810e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:35.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdc77fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdc81017e0 0x7fbdc810e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:35.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdc67fc640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbdc80fc2e0 0x7fbdc810e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:58798/0 (socket says 172.21.15.119:58798) 2024-09-10T19:02:35.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdc67fc640 1 -- 172.21.15.119:0/2027204160 learned_addr learned my addr 172.21.15.119:0/2027204160 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:35.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdc6ffd640 1 --2- 172.21.15.119:0/2027204160 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdc80fb950 0x7fbdc810df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:35.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdc77fe640 1 -- 172.21.15.119:0/2027204160 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbdc80fc2e0 msgr2=0x7fbdc810e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:35.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdc77fe640 1 --2- 172.21.15.119:0/2027204160 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbdc80fc2e0 0x7fbdc810e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:35.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdc77fe640 1 -- 172.21.15.119:0/2027204160 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdc80fb950 msgr2=0x7fbdc810df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:35.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdc77fe640 1 --2- 172.21.15.119:0/2027204160 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdc80fb950 0x7fbdc810df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:35.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdc77fe640 1 -- 172.21.15.119:0/2027204160 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc00098b0 con 0x7fbdc81017e0 2024-09-10T19:02:35.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdc67fc640 1 --2- 172.21.15.119:0/2027204160 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbdc80fc2e0 0x7fbdc810e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:02:35.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdc77fe640 1 --2- 172.21.15.119:0/2027204160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdc81017e0 0x7fbdc810e9d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0001da0 tx=0x7fbdc0004c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:35.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdaf7fe640 1 -- 172.21.15.119:0/2027204160 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc000fdd0 con 0x7fbdc81017e0 2024-09-10T19:02:35.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/2027204160 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdbc005040 con 0x7fbdc81017e0 2024-09-10T19:02:35.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdaf7fe640 1 -- 172.21.15.119:0/2027204160 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fbdc0007d00 con 0x7fbdc81017e0 2024-09-10T19:02:35.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdaf7fe640 1 -- 172.21.15.119:0/2027204160 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0009730 con 0x7fbdc81017e0 2024-09-10T19:02:35.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.978+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/2027204160 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdc8119b20 con 0x7fbdc81017e0 2024-09-10T19:02:35.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.979+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/2027204160 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbdc805de10 con 0x7fbdc81017e0 2024-09-10T19:02:35.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.980+0000 7fbdaf7fe640 1 -- 172.21.15.119:0/2027204160 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7fbdc0007890 con 0x7fbdc81017e0 2024-09-10T19:02:35.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.980+0000 7fbdaf7fe640 1 --2- 172.21.15.119:0/2027204160 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbdb006eed0 0x7fbdb0071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:35.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.980+0000 7fbdc6ffd640 1 --2- 172.21.15.119:0/2027204160 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbdb006eed0 0x7fbdb0071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:35.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.980+0000 7fbdaf7fe640 1 -- 172.21.15.119:0/2027204160 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7fbdc008e4a0 con 0x7fbdc81017e0 2024-09-10T19:02:35.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.981+0000 7fbdaf7fe640 1 --2- 172.21.15.119:0/2027204160 >> v2:172.21.15.194:6804/635822671 conn(0x7fbdb0078780 0x7fbdb007abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:35.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.981+0000 7fbdaf7fe640 1 -- 172.21.15.119:0/2027204160 --> v2:172.21.15.194:6804/635822671 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbdb007b240 con 0x7fbdb0078780 2024-09-10T19:02:35.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.981+0000 7fbdaf7fe640 1 -- 172.21.15.119:0/2027204160 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbdc0090ae0 con 0x7fbdc81017e0 2024-09-10T19:02:35.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.981+0000 7fbdc6ffd640 1 --2- 172.21.15.119:0/2027204160 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbdb006eed0 0x7fbdb0071380 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80041d0 tx=0x7fbdb8004330 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:35.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.981+0000 7fbdc67fc640 1 --2- 172.21.15.119:0/2027204160 >> v2:172.21.15.194:6804/635822671 conn(0x7fbdb0078780 0x7fbdb007abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:35.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.981+0000 7fbdc67fc640 1 --2- 172.21.15.119:0/2027204160 >> v2:172.21.15.194:6804/635822671 conn(0x7fbdb0078780 0x7fbdb007abb0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:35.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.982+0000 7fbdaf7fe640 1 -- 172.21.15.119:0/2027204160 <== osd.5 v2:172.21.15.194:6804/635822671 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fbdb007b240 con 0x7fbdb0078780 2024-09-10T19:02:35.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.994+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/2027204160 --> v2:172.21.15.194:6804/635822671 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbdc80fcf00 con 0x7fbdb0078780 2024-09-10T19:02:35.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.995+0000 7fbdaf7fe640 1 -- 172.21.15.119:0/2027204160 <== osd.5 v2:172.21.15.194:6804/635822671 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fbdc80fcf00 con 0x7fbdb0078780 2024-09-10T19:02:35.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.995+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/2027204160 >> v2:172.21.15.194:6804/635822671 conn(0x7fbdb0078780 msgr2=0x7fbdb007abb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:35.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.995+0000 7fbdcd4e5640 1 --2- 172.21.15.119:0/2027204160 >> v2:172.21.15.194:6804/635822671 conn(0x7fbdb0078780 0x7fbdb007abb0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:35.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.995+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/2027204160 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbdb006eed0 msgr2=0x7fbdb0071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:35.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.995+0000 7fbdcd4e5640 1 --2- 172.21.15.119:0/2027204160 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbdb006eed0 0x7fbdb0071380 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbdb80041d0 tx=0x7fbdb8004330 comp rx=0 tx=0).stop 2024-09-10T19:02:35.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.995+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/2027204160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdc81017e0 msgr2=0x7fbdc810e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:35.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.995+0000 7fbdcd4e5640 1 --2- 172.21.15.119:0/2027204160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbdc81017e0 0x7fbdc810e9d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fbdc0001da0 tx=0x7fbdc0004c50 comp rx=0 tx=0).stop 2024-09-10T19:02:35.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.995+0000 7fbdc77fe640 1 -- 172.21.15.119:0/2027204160 reap_dead start 2024-09-10T19:02:35.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.995+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/2027204160 shutdown_connections 2024-09-10T19:02:35.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.995+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/2027204160 >> 172.21.15.119:0/2027204160 conn(0x7fbdc8072540 msgr2=0x7fbdc80fa5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:35.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.995+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/2027204160 shutdown_connections 2024-09-10T19:02:35.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:34.996+0000 7fbdcd4e5640 1 -- 172.21.15.119:0/2027204160 wait complete. 2024-09-10T19:02:36.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:36 smithi194 ceph-mon[32464]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T19:02:36.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:36 smithi119 ceph-mon[29577]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T19:02:36.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:36 smithi119 ceph-mon[40674]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 16 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T19:02:37.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.572+0000 7fbd42c82640 1 -- 172.21.15.119:0/4090898114 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbd3c0fc1b0 msgr2=0x7fbd3c108580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:37.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.572+0000 7fbd42c82640 1 --2- 172.21.15.119:0/4090898114 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbd3c0fc1b0 0x7fbd3c108580 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009d50 tx=0x7fbd34004c50 comp rx=0 tx=0).stop 2024-09-10T19:02:37.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.572+0000 7fbd42c82640 1 -- 172.21.15.119:0/4090898114 shutdown_connections 2024-09-10T19:02:37.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.572+0000 7fbd42c82640 1 --2- 172.21.15.119:0/4090898114 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbd3c0fc1b0 0x7fbd3c108580 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:37.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.572+0000 7fbd42c82640 1 --2- 172.21.15.119:0/4090898114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd3c0fb800 0x7fbd3c0fbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:37.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.572+0000 7fbd42c82640 1 --2- 172.21.15.119:0/4090898114 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd3c1017d0 0x7fbd3c101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:37.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.572+0000 7fbd42c82640 1 -- 172.21.15.119:0/4090898114 >> 172.21.15.119:0/4090898114 conn(0x7fbd3c071c00 msgr2=0x7fbd3c072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:37.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.572+0000 7fbd42c82640 1 -- 172.21.15.119:0/4090898114 shutdown_connections 2024-09-10T19:02:37.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.572+0000 7fbd42c82640 1 -- 172.21.15.119:0/4090898114 wait complete. 2024-09-10T19:02:37.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.572+0000 7fbd42c82640 1 Processor -- start 2024-09-10T19:02:37.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.572+0000 7fbd42c82640 1 -- start start 2024-09-10T19:02:37.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd42c82640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd3c0fb800 0x7fbd3c1119e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:37.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd42c82640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd3c0fc1b0 0x7fbd3c111f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:37.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd42c82640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbd3c1017d0 0x7fbd3c10bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:37.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd42c82640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3c10c180 con 0x7fbd3c0fb800 2024-09-10T19:02:37.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd42c82640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3c10c2f0 con 0x7fbd3c0fc1b0 2024-09-10T19:02:37.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd42c82640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3c10c4b0 con 0x7fbd3c1017d0 2024-09-10T19:02:37.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd40a1e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd3c0fb800 0x7fbd3c1119e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:37.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd40a1e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd3c0fb800 0x7fbd3c1119e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36768/0 (socket says 172.21.15.119:36768) 2024-09-10T19:02:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd4121f640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbd3c1017d0 0x7fbd3c10bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd40a1e640 1 -- 172.21.15.119:0/283750437 learned_addr learned my addr 172.21.15.119:0/283750437 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd3bfff640 1 --2- 172.21.15.119:0/283750437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd3c0fc1b0 0x7fbd3c111f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd40a1e640 1 -- 172.21.15.119:0/283750437 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbd3c1017d0 msgr2=0x7fbd3c10bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd40a1e640 1 --2- 172.21.15.119:0/283750437 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fbd3c1017d0 0x7fbd3c10bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd40a1e640 1 -- 172.21.15.119:0/283750437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd3c0fc1b0 msgr2=0x7fbd3c111f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd40a1e640 1 --2- 172.21.15.119:0/283750437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd3c0fc1b0 0x7fbd3c111f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd40a1e640 1 -- 172.21.15.119:0/283750437 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2c005040 con 0x7fbd3c0fb800 2024-09-10T19:02:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd40a1e640 1 --2- 172.21.15.119:0/283750437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd3c0fb800 0x7fbd3c1119e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c009550 tx=0x7fbd2c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd397fa640 1 -- 172.21.15.119:0/283750437 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c00e8b0 con 0x7fbd3c0fb800 2024-09-10T19:02:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.573+0000 7fbd42c82640 1 -- 172.21.15.119:0/283750437 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd340098b0 con 0x7fbd3c0fb800 2024-09-10T19:02:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.574+0000 7fbd397fa640 1 -- 172.21.15.119:0/283750437 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fbd2c00f040 con 0x7fbd3c0fb800 2024-09-10T19:02:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.574+0000 7fbd42c82640 1 -- 172.21.15.119:0/283750437 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd3c118870 con 0x7fbd3c0fb800 2024-09-10T19:02:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.574+0000 7fbd397fa640 1 -- 172.21.15.119:0/283750437 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2c013b30 con 0x7fbd3c0fb800 2024-09-10T19:02:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.574+0000 7fbd42c82640 1 -- 172.21.15.119:0/283750437 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbd3c0fa680 con 0x7fbd3c0fb800 2024-09-10T19:02:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.575+0000 7fbd397fa640 1 -- 172.21.15.119:0/283750437 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7fbd2c00ea10 con 0x7fbd3c0fb800 2024-09-10T19:02:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.575+0000 7fbd397fa640 1 --2- 172.21.15.119:0/283750437 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbd2806eed0 0x7fbd28071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.575+0000 7fbd397fa640 1 -- 172.21.15.119:0/283750437 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7fbd2c08e300 con 0x7fbd3c0fb800 2024-09-10T19:02:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.575+0000 7fbd3bfff640 1 --2- 172.21.15.119:0/283750437 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbd2806eed0 0x7fbd28071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.576+0000 7fbd397fa640 1 --2- 172.21.15.119:0/283750437 >> v2:172.21.15.194:6808/2830881352 conn(0x7fbd28078780 0x7fbd2807abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.576+0000 7fbd397fa640 1 -- 172.21.15.119:0/283750437 --> v2:172.21.15.194:6808/2830881352 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbd2807b240 con 0x7fbd28078780 2024-09-10T19:02:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.576+0000 7fbd397fa640 1 -- 172.21.15.119:0/283750437 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbd2c090940 con 0x7fbd3c0fb800 2024-09-10T19:02:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.576+0000 7fbd3bfff640 1 --2- 172.21.15.119:0/283750437 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbd2806eed0 0x7fbd28071380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c10d2e0 tx=0x7fbd24009040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.576+0000 7fbd4121f640 1 --2- 172.21.15.119:0/283750437 >> v2:172.21.15.194:6808/2830881352 conn(0x7fbd28078780 0x7fbd2807abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.576+0000 7fbd4121f640 1 --2- 172.21.15.119:0/283750437 >> v2:172.21.15.194:6808/2830881352 conn(0x7fbd28078780 0x7fbd2807abb0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.577+0000 7fbd397fa640 1 -- 172.21.15.119:0/283750437 <== osd.6 v2:172.21.15.194:6808/2830881352 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fbd2807b240 con 0x7fbd28078780 2024-09-10T19:02:37.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.590+0000 7fbd42c82640 1 -- 172.21.15.119:0/283750437 --> v2:172.21.15.194:6808/2830881352 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbd3c0fcef0 con 0x7fbd28078780 2024-09-10T19:02:37.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.590+0000 7fbd397fa640 1 -- 172.21.15.119:0/283750437 <== osd.6 v2:172.21.15.194:6808/2830881352 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fbd3c0fcef0 con 0x7fbd28078780 2024-09-10T19:02:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.591+0000 7fbd42c82640 1 -- 172.21.15.119:0/283750437 >> v2:172.21.15.194:6808/2830881352 conn(0x7fbd28078780 msgr2=0x7fbd2807abb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.591+0000 7fbd42c82640 1 --2- 172.21.15.119:0/283750437 >> v2:172.21.15.194:6808/2830881352 conn(0x7fbd28078780 0x7fbd2807abb0 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.591+0000 7fbd42c82640 1 -- 172.21.15.119:0/283750437 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbd2806eed0 msgr2=0x7fbd28071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.591+0000 7fbd42c82640 1 --2- 172.21.15.119:0/283750437 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbd2806eed0 0x7fbd28071380 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c10d2e0 tx=0x7fbd24009040 comp rx=0 tx=0).stop 2024-09-10T19:02:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.591+0000 7fbd42c82640 1 -- 172.21.15.119:0/283750437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd3c0fb800 msgr2=0x7fbd3c1119e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.591+0000 7fbd42c82640 1 --2- 172.21.15.119:0/283750437 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd3c0fb800 0x7fbd3c1119e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fbd2c009550 tx=0x7fbd2c009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.591+0000 7fbd4121f640 1 -- 172.21.15.119:0/283750437 reap_dead start 2024-09-10T19:02:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.591+0000 7fbd42c82640 1 -- 172.21.15.119:0/283750437 shutdown_connections 2024-09-10T19:02:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.591+0000 7fbd42c82640 1 -- 172.21.15.119:0/283750437 >> 172.21.15.119:0/283750437 conn(0x7fbd3c071c00 msgr2=0x7fbd3c0f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.591+0000 7fbd42c82640 1 -- 172.21.15.119:0/283750437 shutdown_connections 2024-09-10T19:02:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:37.591+0000 7fbd42c82640 1 -- 172.21.15.119:0/283750437 wait complete. 2024-09-10T19:02:38.113 INFO:teuthology.orchestra.run.smithi119.stdout:77309411367 2024-09-10T19:02:38.113 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-10T19:02:38.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:38 smithi194 ceph-mon[32464]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:38.519 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:02:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:02:38.533 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:38 smithi119 ceph-mon[29577]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:38.533 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:38 smithi119 ceph-mon[40674]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:38.917 INFO:teuthology.orchestra.run.smithi119.stdout:154618822681 2024-09-10T19:02:38.917 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-10T19:02:39.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:39 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:39.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:39 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:39.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:39 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:39.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.515+0000 7f2b8387c640 1 -- 172.21.15.119:0/3275446780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b7c0fd8e0 msgr2=0x7f2b7c0fdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:39.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.515+0000 7f2b8387c640 1 --2- 172.21.15.119:0/3275446780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b7c0fd8e0 0x7f2b7c0fdcb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2b78004890 tx=0x7f2b78009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:39.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.515+0000 7f2b8387c640 1 -- 172.21.15.119:0/3275446780 shutdown_connections 2024-09-10T19:02:39.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.515+0000 7f2b8387c640 1 --2- 172.21.15.119:0/3275446780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b7c106e90 0x7f2b7c109270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:39.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.515+0000 7f2b8387c640 1 --2- 172.21.15.119:0/3275446780 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b7c0fe280 0x7f2b7c106950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:39.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.515+0000 7f2b8387c640 1 --2- 172.21.15.119:0/3275446780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b7c0fd8e0 0x7f2b7c0fdcb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:39.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.515+0000 7f2b8387c640 1 -- 172.21.15.119:0/3275446780 >> 172.21.15.119:0/3275446780 conn(0x7f2b7c071c00 msgr2=0x7f2b7c072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:39.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.515+0000 7f2b8387c640 1 -- 172.21.15.119:0/3275446780 shutdown_connections 2024-09-10T19:02:39.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.515+0000 7f2b8387c640 1 -- 172.21.15.119:0/3275446780 wait complete. 2024-09-10T19:02:39.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b8387c640 1 Processor -- start 2024-09-10T19:02:39.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b8387c640 1 -- start start 2024-09-10T19:02:39.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b8387c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b7c0fd8e0 0x7f2b7c112e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:39.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b8387c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b7c0fe280 0x7f2b7c113390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:39.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b8387c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b7c106e90 0x7f2b7c10cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:39.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b8387c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c10d630 con 0x7f2b7c0fe280 2024-09-10T19:02:39.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b8387c640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c10d7a0 con 0x7f2b7c0fd8e0 2024-09-10T19:02:39.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b8387c640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b7c10d960 con 0x7f2b7c106e90 2024-09-10T19:02:39.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b80e17640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b7c0fe280 0x7f2b7c113390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:39.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b81e19640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b7c106e90 0x7f2b7c10cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:39.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b80e17640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b7c0fe280 0x7f2b7c113390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36790/0 (socket says 172.21.15.119:36790) 2024-09-10T19:02:39.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b80e17640 1 -- 172.21.15.119:0/160163158 learned_addr learned my addr 172.21.15.119:0/160163158 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:39.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b81618640 1 --2- 172.21.15.119:0/160163158 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b7c0fd8e0 0x7f2b7c112e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:39.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b81e19640 1 -- 172.21.15.119:0/160163158 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b7c0fd8e0 msgr2=0x7f2b7c112e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:39.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b81e19640 1 --2- 172.21.15.119:0/160163158 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b7c0fd8e0 0x7f2b7c112e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:39.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b81e19640 1 -- 172.21.15.119:0/160163158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b7c0fe280 msgr2=0x7f2b7c113390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:39.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b81e19640 1 --2- 172.21.15.119:0/160163158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b7c0fe280 0x7f2b7c113390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:39.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b81e19640 1 -- 172.21.15.119:0/160163158 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b6c005040 con 0x7f2b7c106e90 2024-09-10T19:02:39.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.516+0000 7f2b80e17640 1 --2- 172.21.15.119:0/160163158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b7c0fe280 0x7f2b7c113390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:02:39.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.517+0000 7f2b81e19640 1 --2- 172.21.15.119:0/160163158 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b7c106e90 0x7f2b7c10cfe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c067030 tx=0x7f2b74004db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:39.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.517+0000 7f2b71ffb640 1 -- 172.21.15.119:0/160163158 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b74017070 con 0x7f2b7c106e90 2024-09-10T19:02:39.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.517+0000 7f2b71ffb640 1 -- 172.21.15.119:0/160163158 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f2b7400f040 con 0x7f2b7c106e90 2024-09-10T19:02:39.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.517+0000 7f2b8387c640 1 -- 172.21.15.119:0/160163158 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b78005040 con 0x7f2b7c106e90 2024-09-10T19:02:39.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.517+0000 7f2b71ffb640 1 -- 172.21.15.119:0/160163158 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b740036a0 con 0x7f2b7c106e90 2024-09-10T19:02:39.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.517+0000 7f2b8387c640 1 -- 172.21.15.119:0/160163158 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b7c119ca0 con 0x7f2b7c106e90 2024-09-10T19:02:39.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.518+0000 7f2b8387c640 1 -- 172.21.15.119:0/160163158 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2b7c05ddd0 con 0x7f2b7c106e90 2024-09-10T19:02:39.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.518+0000 7f2b71ffb640 1 -- 172.21.15.119:0/160163158 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f2b74003ad0 con 0x7f2b7c106e90 2024-09-10T19:02:39.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.519+0000 7f2b71ffb640 1 --2- 172.21.15.119:0/160163158 >> v2:172.21.15.119:6800/3118961733 conn(0x7f2b6806eed0 0x7f2b68071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:39.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.519+0000 7f2b71ffb640 1 -- 172.21.15.119:0/160163158 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f2b7408e410 con 0x7f2b7c106e90 2024-09-10T19:02:39.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.519+0000 7f2b81618640 1 --2- 172.21.15.119:0/160163158 >> v2:172.21.15.119:6800/3118961733 conn(0x7f2b6806eed0 0x7f2b68071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:39.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.519+0000 7f2b71ffb640 1 --2- 172.21.15.119:0/160163158 >> v2:172.21.15.119:6801/1598455131 conn(0x7f2b68078780 0x7f2b6807abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:39.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.519+0000 7f2b71ffb640 1 -- 172.21.15.119:0/160163158 --> v2:172.21.15.119:6801/1598455131 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2b6807b240 con 0x7f2b68078780 2024-09-10T19:02:39.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.519+0000 7f2b71ffb640 1 -- 172.21.15.119:0/160163158 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2b74090a50 con 0x7f2b7c106e90 2024-09-10T19:02:39.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.519+0000 7f2b80e17640 1 --2- 172.21.15.119:0/160163158 >> v2:172.21.15.119:6801/1598455131 conn(0x7f2b68078780 0x7f2b6807abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:39.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.519+0000 7f2b81618640 1 --2- 172.21.15.119:0/160163158 >> v2:172.21.15.119:6800/3118961733 conn(0x7f2b6806eed0 0x7f2b68071380 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009040 tx=0x7f2b78009da0 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:39.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.519+0000 7f2b80e17640 1 --2- 172.21.15.119:0/160163158 >> v2:172.21.15.119:6801/1598455131 conn(0x7f2b68078780 0x7f2b6807abb0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:39.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.520+0000 7f2b71ffb640 1 -- 172.21.15.119:0/160163158 <== osd.0 v2:172.21.15.119:6801/1598455131 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f2b6807b240 con 0x7f2b68078780 2024-09-10T19:02:39.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.532+0000 7f2b8387c640 1 -- 172.21.15.119:0/160163158 --> v2:172.21.15.119:6801/1598455131 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2b7c065a60 con 0x7f2b68078780 2024-09-10T19:02:39.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.532+0000 7f2b71ffb640 1 -- 172.21.15.119:0/160163158 <== osd.0 v2:172.21.15.119:6801/1598455131 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2b7c065a60 con 0x7f2b68078780 2024-09-10T19:02:39.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.532+0000 7f2b8387c640 1 -- 172.21.15.119:0/160163158 >> v2:172.21.15.119:6801/1598455131 conn(0x7f2b68078780 msgr2=0x7f2b6807abb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:39.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.533+0000 7f2b8387c640 1 --2- 172.21.15.119:0/160163158 >> v2:172.21.15.119:6801/1598455131 conn(0x7f2b68078780 0x7f2b6807abb0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:39.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.533+0000 7f2b8387c640 1 -- 172.21.15.119:0/160163158 >> v2:172.21.15.119:6800/3118961733 conn(0x7f2b6806eed0 msgr2=0x7f2b68071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:39.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.533+0000 7f2b8387c640 1 --2- 172.21.15.119:0/160163158 >> v2:172.21.15.119:6800/3118961733 conn(0x7f2b6806eed0 0x7f2b68071380 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2b78009040 tx=0x7f2b78009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:39.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.533+0000 7f2b8387c640 1 -- 172.21.15.119:0/160163158 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b7c106e90 msgr2=0x7f2b7c10cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:39.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.533+0000 7f2b8387c640 1 --2- 172.21.15.119:0/160163158 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b7c106e90 0x7f2b7c10cfe0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2b7c067030 tx=0x7f2b74004db0 comp rx=0 tx=0).stop 2024-09-10T19:02:39.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.533+0000 7f2b81e19640 1 -- 172.21.15.119:0/160163158 reap_dead start 2024-09-10T19:02:39.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.533+0000 7f2b8387c640 1 -- 172.21.15.119:0/160163158 shutdown_connections 2024-09-10T19:02:39.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.533+0000 7f2b8387c640 1 -- 172.21.15.119:0/160163158 >> 172.21.15.119:0/160163158 conn(0x7f2b7c071c00 msgr2=0x7f2b7c1075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:39.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.533+0000 7f2b8387c640 1 -- 172.21.15.119:0/160163158 shutdown_connections 2024-09-10T19:02:39.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:39.533+0000 7f2b8387c640 1 -- 172.21.15.119:0/160163158 wait complete. 2024-09-10T19:02:40.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:40 smithi119 ceph-mon[29577]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:40.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:40 smithi119 ceph-mon[40674]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:40.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:40 smithi194 ceph-mon[32464]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:40.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:02:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:02:41.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.060+0000 7f40aa9f8640 1 -- 172.21.15.119:0/713539444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a4102870 msgr2=0x7f40a4102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:41.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.060+0000 7f40aa9f8640 1 --2- 172.21.15.119:0/713539444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a4102870 0x7f40a4102c40 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4094004950 tx=0x7f4094009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:41.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.060+0000 7f40aa9f8640 1 -- 172.21.15.119:0/713539444 shutdown_connections 2024-09-10T19:02:41.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.060+0000 7f40aa9f8640 1 --2- 172.21.15.119:0/713539444 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40a40fb530 0x7f40a4109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:41.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.060+0000 7f40aa9f8640 1 --2- 172.21.15.119:0/713539444 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f40a4103210 0x7f40a40fae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:41.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.060+0000 7f40aa9f8640 1 --2- 172.21.15.119:0/713539444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a4102870 0x7f40a4102c40 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:41.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.060+0000 7f40aa9f8640 1 -- 172.21.15.119:0/713539444 >> 172.21.15.119:0/713539444 conn(0x7f40a4074d90 msgr2=0x7f40a40f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:41.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.060+0000 7f40aa9f8640 1 -- 172.21.15.119:0/713539444 shutdown_connections 2024-09-10T19:02:41.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.060+0000 7f40aa9f8640 1 -- 172.21.15.119:0/713539444 wait complete. 2024-09-10T19:02:41.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.060+0000 7f40aa9f8640 1 Processor -- start 2024-09-10T19:02:41.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40aa9f8640 1 -- start start 2024-09-10T19:02:41.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40aa9f8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a40fb530 0x7f40a4112dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:41.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40aa9f8640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f40a4102870 0x7f40a4113310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40aa9f8640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40a4103210 0x7f40a410cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40aa9f8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a410d5b0 con 0x7f40a40fb530 2024-09-10T19:02:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40aa9f8640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a410d720 con 0x7f40a4103210 2024-09-10T19:02:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40aa9f8640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a410d8e0 con 0x7f40a4102870 2024-09-10T19:02:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40a3fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a40fb530 0x7f40a4112dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40a37fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f40a4102870 0x7f40a4113310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40a3fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a40fb530 0x7f40a4112dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36806/0 (socket says 172.21.15.119:36806) 2024-09-10T19:02:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40a3fff640 1 -- 172.21.15.119:0/3660296239 learned_addr learned my addr 172.21.15.119:0/3660296239 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40a3fff640 1 -- 172.21.15.119:0/3660296239 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f40a4102870 msgr2=0x7f40a4113310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40a3fff640 1 --2- 172.21.15.119:0/3660296239 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f40a4102870 0x7f40a4113310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40a3fff640 1 -- 172.21.15.119:0/3660296239 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40a4103210 msgr2=0x7f40a410cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40a8f95640 1 --2- 172.21.15.119:0/3660296239 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40a4103210 0x7f40a410cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40a3fff640 1 --2- 172.21.15.119:0/3660296239 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40a4103210 0x7f40a410cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40a3fff640 1 -- 172.21.15.119:0/3660296239 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c005040 con 0x7f40a40fb530 2024-09-10T19:02:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40a8f95640 1 --2- 172.21.15.119:0/3660296239 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40a4103210 0x7f40a410cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T19:02:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40a3fff640 1 --2- 172.21.15.119:0/3660296239 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a40fb530 0x7f40a4112dd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f409402d040 tx=0x7f4094035040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.061+0000 7f40a0ff9640 1 -- 172.21.15.119:0/3660296239 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4094009850 con 0x7f40a40fb530 2024-09-10T19:02:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.062+0000 7f40aa9f8640 1 -- 172.21.15.119:0/3660296239 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4094005040 con 0x7f40a40fb530 2024-09-10T19:02:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.062+0000 7f40a0ff9640 1 -- 172.21.15.119:0/3660296239 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f409402ab70 con 0x7f40a40fb530 2024-09-10T19:02:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.062+0000 7f40a0ff9640 1 -- 172.21.15.119:0/3660296239 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4094033420 con 0x7f40a40fb530 2024-09-10T19:02:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.062+0000 7f40aa9f8640 1 -- 172.21.15.119:0/3660296239 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a4119c20 con 0x7f40a40fb530 2024-09-10T19:02:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.063+0000 7f40aa9f8640 1 -- 172.21.15.119:0/3660296239 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f40a4101df0 con 0x7f40a40fb530 2024-09-10T19:02:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.063+0000 7f40a0ff9640 1 -- 172.21.15.119:0/3660296239 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f4094030020 con 0x7f40a40fb530 2024-09-10T19:02:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.063+0000 7f40a0ff9640 1 --2- 172.21.15.119:0/3660296239 >> v2:172.21.15.119:6800/3118961733 conn(0x7f409006eed0 0x7f4090071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:41.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.064+0000 7f40a0ff9640 1 -- 172.21.15.119:0/3660296239 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f40940ae830 con 0x7f40a40fb530 2024-09-10T19:02:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.064+0000 7f40a37fe640 1 --2- 172.21.15.119:0/3660296239 >> v2:172.21.15.119:6800/3118961733 conn(0x7f409006eed0 0x7f4090071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.064+0000 7f40a0ff9640 1 --2- 172.21.15.119:0/3660296239 >> v2:172.21.15.119:6813/783850966 conn(0x7f4090078780 0x7f409007abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.064+0000 7f40a0ff9640 1 -- 172.21.15.119:0/3660296239 --> v2:172.21.15.119:6813/783850966 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f409007b240 con 0x7f4090078780 2024-09-10T19:02:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.064+0000 7f40a0ff9640 1 -- 172.21.15.119:0/3660296239 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40940b3240 con 0x7f40a40fb530 2024-09-10T19:02:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.064+0000 7f40a8f95640 1 --2- 172.21.15.119:0/3660296239 >> v2:172.21.15.119:6813/783850966 conn(0x7f4090078780 0x7f409007abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.064+0000 7f40a37fe640 1 --2- 172.21.15.119:0/3660296239 >> v2:172.21.15.119:6800/3118961733 conn(0x7f409006eed0 0x7f4090071380 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f40a410e690 tx=0x7f408c00b040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.064+0000 7f40a8f95640 1 --2- 172.21.15.119:0/3660296239 >> v2:172.21.15.119:6813/783850966 conn(0x7f4090078780 0x7f409007abb0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.065+0000 7f40a0ff9640 1 -- 172.21.15.119:0/3660296239 <== osd.3 v2:172.21.15.119:6813/783850966 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f409007b240 con 0x7f4090078780 2024-09-10T19:02:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.077+0000 7f40aa9f8640 1 -- 172.21.15.119:0/3660296239 --> v2:172.21.15.119:6813/783850966 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f40a40fe120 con 0x7f4090078780 2024-09-10T19:02:41.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.077+0000 7f40a0ff9640 1 -- 172.21.15.119:0/3660296239 <== osd.3 v2:172.21.15.119:6813/783850966 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f40a40fe120 con 0x7f4090078780 2024-09-10T19:02:41.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.078+0000 7f40aa9f8640 1 -- 172.21.15.119:0/3660296239 >> v2:172.21.15.119:6813/783850966 conn(0x7f4090078780 msgr2=0x7f409007abb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:41.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.078+0000 7f40aa9f8640 1 --2- 172.21.15.119:0/3660296239 >> v2:172.21.15.119:6813/783850966 conn(0x7f4090078780 0x7f409007abb0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:41.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.078+0000 7f40aa9f8640 1 -- 172.21.15.119:0/3660296239 >> v2:172.21.15.119:6800/3118961733 conn(0x7f409006eed0 msgr2=0x7f4090071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:41.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.078+0000 7f40aa9f8640 1 --2- 172.21.15.119:0/3660296239 >> v2:172.21.15.119:6800/3118961733 conn(0x7f409006eed0 0x7f4090071380 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f40a410e690 tx=0x7f408c00b040 comp rx=0 tx=0).stop 2024-09-10T19:02:41.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.078+0000 7f40aa9f8640 1 -- 172.21.15.119:0/3660296239 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a40fb530 msgr2=0x7f40a4112dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:41.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.078+0000 7f40aa9f8640 1 --2- 172.21.15.119:0/3660296239 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a40fb530 0x7f40a4112dd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f409402d040 tx=0x7f4094035040 comp rx=0 tx=0).stop 2024-09-10T19:02:41.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.078+0000 7f40a8f95640 1 -- 172.21.15.119:0/3660296239 reap_dead start 2024-09-10T19:02:41.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.078+0000 7f40aa9f8640 1 -- 172.21.15.119:0/3660296239 shutdown_connections 2024-09-10T19:02:41.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.078+0000 7f40aa9f8640 1 -- 172.21.15.119:0/3660296239 >> 172.21.15.119:0/3660296239 conn(0x7f40a4074d90 msgr2=0x7f40a4100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:41.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.078+0000 7f40aa9f8640 1 -- 172.21.15.119:0/3660296239 shutdown_connections 2024-09-10T19:02:41.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:41.078+0000 7f40aa9f8640 1 -- 172.21.15.119:0/3660296239 wait complete. 2024-09-10T19:02:41.860 INFO:teuthology.orchestra.run.smithi119.stdout:180388626454 2024-09-10T19:02:41.861 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-10T19:02:42.377 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:42 smithi119 ceph-mon[29577]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:02:42.377 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:42 smithi119 ceph-mon[40674]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:02:42.378 INFO:teuthology.orchestra.run.smithi119.stdout:107374182436 2024-09-10T19:02:42.378 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-10T19:02:42.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:42 smithi194 ceph-mon[32464]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:02:44.069 INFO:teuthology.orchestra.run.smithi119.stdout:34359738417 2024-09-10T19:02:44.069 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-10T19:02:44.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:44 smithi119 ceph-mon[40674]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:44.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:44 smithi119 ceph-mon[29577]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:44.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:44 smithi194 ceph-mon[32464]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:44.683 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:45.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.987+0000 7fc01a99a640 1 -- 172.21.15.119:0/1433498638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0140fd8c0 msgr2=0x7fc0140fdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:45.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.987+0000 7fc01a99a640 1 --2- 172.21.15.119:0/1433498638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0140fd8c0 0x7fc0140fdc90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc004004950 tx=0x7fc004009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:45.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.987+0000 7fc01a99a640 1 -- 172.21.15.119:0/1433498638 shutdown_connections 2024-09-10T19:02:45.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.987+0000 7fc01a99a640 1 --2- 172.21.15.119:0/1433498638 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc014106e70 0x7fc014109250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:45.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.987+0000 7fc01a99a640 1 --2- 172.21.15.119:0/1433498638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0140fe260 0x7fc014106930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:45.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.987+0000 7fc01a99a640 1 --2- 172.21.15.119:0/1433498638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0140fd8c0 0x7fc0140fdc90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:45.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.987+0000 7fc01a99a640 1 -- 172.21.15.119:0/1433498638 >> 172.21.15.119:0/1433498638 conn(0x7fc014071c00 msgr2=0x7fc014072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:45.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.987+0000 7fc01a99a640 1 -- 172.21.15.119:0/1433498638 shutdown_connections 2024-09-10T19:02:45.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.987+0000 7fc01a99a640 1 -- 172.21.15.119:0/1433498638 wait complete. 2024-09-10T19:02:45.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.988+0000 7fc01a99a640 1 Processor -- start 2024-09-10T19:02:45.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.988+0000 7fc01a99a640 1 -- start start 2024-09-10T19:02:45.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.988+0000 7fc01a99a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc0140fd8c0 0x7fc014112da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:45.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.988+0000 7fc01a99a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0140fe260 0x7fc0141132e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:45.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.988+0000 7fc01a99a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc014106e70 0x7fc01410cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:45.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.988+0000 7fc01a99a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01410d5d0 con 0x7fc014106e70 2024-09-10T19:02:45.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.988+0000 7fc01a99a640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01410d740 con 0x7fc0140fe260 2024-09-10T19:02:45.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.988+0000 7fc01a99a640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc01410d900 con 0x7fc0140fd8c0 2024-09-10T19:02:45.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.988+0000 7fc013fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc0140fd8c0 0x7fc014112da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:45.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.988+0000 7fc013fff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc0140fd8c0 0x7fc014112da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:48794/0 (socket says 172.21.15.119:48794) 2024-09-10T19:02:45.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.988+0000 7fc018f37640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc014106e70 0x7fc01410cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:45.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.988+0000 7fc013fff640 1 -- 172.21.15.119:0/3599350271 learned_addr learned my addr 172.21.15.119:0/3599350271 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:45.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.988+0000 7fc0137fe640 1 --2- 172.21.15.119:0/3599350271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0140fe260 0x7fc0141132e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:45.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.989+0000 7fc018f37640 1 -- 172.21.15.119:0/3599350271 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc0140fd8c0 msgr2=0x7fc014112da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:45.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.989+0000 7fc018f37640 1 --2- 172.21.15.119:0/3599350271 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc0140fd8c0 0x7fc014112da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:45.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.989+0000 7fc018f37640 1 -- 172.21.15.119:0/3599350271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0140fe260 msgr2=0x7fc0141132e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:45.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.989+0000 7fc018f37640 1 --2- 172.21.15.119:0/3599350271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc0140fe260 0x7fc0141132e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:45.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.989+0000 7fc018f37640 1 -- 172.21.15.119:0/3599350271 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc004005040 con 0x7fc014106e70 2024-09-10T19:02:45.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.989+0000 7fc013fff640 1 --2- 172.21.15.119:0/3599350271 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc0140fd8c0 0x7fc014112da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:02:45.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.989+0000 7fc018f37640 1 --2- 172.21.15.119:0/3599350271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc014106e70 0x7fc01410cf80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00eba0 tx=0x7fc00c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:45.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.989+0000 7fc010ff9640 1 -- 172.21.15.119:0/3599350271 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c004020 con 0x7fc014106e70 2024-09-10T19:02:45.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.989+0000 7fc010ff9640 1 -- 172.21.15.119:0/3599350271 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fc00c00f040 con 0x7fc014106e70 2024-09-10T19:02:45.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.989+0000 7fc01a99a640 1 -- 172.21.15.119:0/3599350271 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc01410dbe0 con 0x7fc014106e70 2024-09-10T19:02:45.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.989+0000 7fc010ff9640 1 -- 172.21.15.119:0/3599350271 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c00cdd0 con 0x7fc014106e70 2024-09-10T19:02:45.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.989+0000 7fc01a99a640 1 -- 172.21.15.119:0/3599350271 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc014119d90 con 0x7fc014106e70 2024-09-10T19:02:45.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.990+0000 7fc01a99a640 1 -- 172.21.15.119:0/3599350271 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc014065a80 con 0x7fc014106e70 2024-09-10T19:02:45.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.991+0000 7fc010ff9640 1 -- 172.21.15.119:0/3599350271 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7fc00c007aa0 con 0x7fc014106e70 2024-09-10T19:02:45.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.991+0000 7fc010ff9640 1 --2- 172.21.15.119:0/3599350271 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbffc06eed0 0x7fbffc071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:45.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.991+0000 7fc013fff640 1 --2- 172.21.15.119:0/3599350271 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbffc06eed0 0x7fbffc071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:45.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.991+0000 7fc010ff9640 1 -- 172.21.15.119:0/3599350271 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7fc00c08f160 con 0x7fc014106e70 2024-09-10T19:02:45.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.991+0000 7fc010ff9640 1 --2- 172.21.15.119:0/3599350271 >> v2:172.21.15.194:6800/4259870591 conn(0x7fbffc078800 0x7fbffc07ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:45.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.991+0000 7fc010ff9640 1 -- 172.21.15.119:0/3599350271 --> v2:172.21.15.194:6800/4259870591 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbffc07b2c0 con 0x7fbffc078800 2024-09-10T19:02:45.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.991+0000 7fc010ff9640 1 -- 172.21.15.119:0/3599350271 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=61) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc00c08f530 con 0x7fc014106e70 2024-09-10T19:02:45.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.992+0000 7fc0137fe640 1 --2- 172.21.15.119:0/3599350271 >> v2:172.21.15.194:6800/4259870591 conn(0x7fbffc078800 0x7fbffc07ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:45.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.992+0000 7fc013fff640 1 --2- 172.21.15.119:0/3599350271 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbffc06eed0 0x7fbffc071380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc004004540 tx=0x7fc004035040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:45.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.992+0000 7fc0137fe640 1 --2- 172.21.15.119:0/3599350271 >> v2:172.21.15.194:6800/4259870591 conn(0x7fbffc078800 0x7fbffc07ac30 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:45.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:44.993+0000 7fc010ff9640 1 -- 172.21.15.119:0/3599350271 <== osd.4 v2:172.21.15.194:6800/4259870591 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fbffc07b2c0 con 0x7fbffc078800 2024-09-10T19:02:45.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:45.005+0000 7fc01a99a640 1 -- 172.21.15.119:0/3599350271 --> v2:172.21.15.194:6800/4259870591 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc01405dd90 con 0x7fbffc078800 2024-09-10T19:02:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:45.005+0000 7fc010ff9640 1 -- 172.21.15.119:0/3599350271 <== osd.4 v2:172.21.15.194:6800/4259870591 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc01405dd90 con 0x7fbffc078800 2024-09-10T19:02:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:45.006+0000 7fc01a99a640 1 -- 172.21.15.119:0/3599350271 >> v2:172.21.15.194:6800/4259870591 conn(0x7fbffc078800 msgr2=0x7fbffc07ac30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:45.006+0000 7fc01a99a640 1 --2- 172.21.15.119:0/3599350271 >> v2:172.21.15.194:6800/4259870591 conn(0x7fbffc078800 0x7fbffc07ac30 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:45.006+0000 7fc01a99a640 1 -- 172.21.15.119:0/3599350271 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbffc06eed0 msgr2=0x7fbffc071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:45.006+0000 7fc01a99a640 1 --2- 172.21.15.119:0/3599350271 >> v2:172.21.15.119:6800/3118961733 conn(0x7fbffc06eed0 0x7fbffc071380 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc004004540 tx=0x7fc004035040 comp rx=0 tx=0).stop 2024-09-10T19:02:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:45.006+0000 7fc01a99a640 1 -- 172.21.15.119:0/3599350271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc014106e70 msgr2=0x7fc01410cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:45.006+0000 7fc01a99a640 1 --2- 172.21.15.119:0/3599350271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc014106e70 0x7fc01410cf80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc00c00eba0 tx=0x7fc00c00b040 comp rx=0 tx=0).stop 2024-09-10T19:02:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:45.006+0000 7fc018f37640 1 -- 172.21.15.119:0/3599350271 reap_dead start 2024-09-10T19:02:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:45.006+0000 7fc01a99a640 1 -- 172.21.15.119:0/3599350271 shutdown_connections 2024-09-10T19:02:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:45.006+0000 7fc01a99a640 1 -- 172.21.15.119:0/3599350271 >> 172.21.15.119:0/3599350271 conn(0x7fc014071c00 msgr2=0x7fc014107580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:45.006+0000 7fc01a99a640 1 -- 172.21.15.119:0/3599350271 shutdown_connections 2024-09-10T19:02:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:45.006+0000 7fc01a99a640 1 -- 172.21.15.119:0/3599350271 wait complete. 2024-09-10T19:02:45.253 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:45.255 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:45.260 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:45.317 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:45 smithi119 systemd[1]: Starting Ceph alertmanager.a for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:02:45.928 INFO:teuthology.orchestra.run.smithi119.stdout:128849018912 2024-09-10T19:02:45.928 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-10T19:02:46.173 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:46 smithi119 ceph-mon[29577]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:46.173 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:46 smithi119 ceph-mon[40674]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:46.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:46 smithi194 ceph-mon[32464]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:46.595 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:47.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:47.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:47.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:02:48.448 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:48 smithi119 ceph-mon[29577]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:48.448 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:48 smithi119 ceph-mon[40674]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:48.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:48 smithi194 ceph-mon[32464]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:48.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:02:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:02:49.296 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:49 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:49.296 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:49 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:49.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:49 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:50.006 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:50.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:50 smithi119 ceph-mon[29577]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:50.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:50 smithi119 ceph-mon[40674]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:50.413 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:02:50.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:50 smithi194 ceph-mon[32464]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:50.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:02:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:02:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:02:50.658 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:50 smithi119 podman[75284]: 2024-09-10 19:02:50.553349216 +0000 UTC m=+0.125190500 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-10T19:02:51.316 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:51 smithi119 podman[75284]: 2024-09-10 19:02:50.999930408 +0000 UTC m=+0.571771672 volume create fe8babc2ce04fdc80c79235e44755d2e33bc4dbe12d8661ba6d9b2ccab824db8 2024-09-10T19:02:51.317 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:51 smithi119 podman[75284]: 2024-09-10 19:02:51.117309708 +0000 UTC m=+0.689150972 container create 7e27161edb5ac44951c597d1d3dcfa55918121ee65e7b899879936c3881457b1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:02:51.715 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:51 smithi119 podman[75284]: 2024-09-10 19:02:51.54624041 +0000 UTC m=+1.118081677 container init 7e27161edb5ac44951c597d1d3dcfa55918121ee65e7b899879936c3881457b1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:02:51.715 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:51 smithi119 podman[75284]: 2024-09-10 19:02:51.551310276 +0000 UTC m=+1.123151541 container start 7e27161edb5ac44951c597d1d3dcfa55918121ee65e7b899879936c3881457b1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:02:51.715 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:51 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[75363]: ts=2024-09-10T19:02:51.568Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-10T19:02:51.715 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:51 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[75363]: ts=2024-09-10T19:02:51.568Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-10T19:02:51.716 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:51 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[75363]: ts=2024-09-10T19:02:51.569Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.119 port=9094 2024-09-10T19:02:51.716 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:51 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[75363]: ts=2024-09-10T19:02:51.571Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-10T19:02:51.716 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:51 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[75363]: ts=2024-09-10T19:02:51.598Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-10T19:02:51.716 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:51 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[75363]: ts=2024-09-10T19:02:51.598Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-10T19:02:51.716 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:51 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[75363]: ts=2024-09-10T19:02:51.601Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-10T19:02:51.716 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:51 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[75363]: ts=2024-09-10T19:02:51.601Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-10T19:02:51.716 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:51 smithi119 bash[75284]: 7e27161edb5ac44951c597d1d3dcfa55918121ee65e7b899879936c3881457b1 2024-09-10T19:02:51.716 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:51 smithi119 systemd[1]: Started Ceph alertmanager.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:02:51.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.875+0000 7f83cc612640 1 -- 172.21.15.119:0/1529274305 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83c40fd8c0 msgr2=0x7f83c40fdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:51.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.875+0000 7f83cc612640 1 --2- 172.21.15.119:0/1529274305 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83c40fd8c0 0x7f83c40fdc90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f83bc004890 tx=0x7f83bc009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:51.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.875+0000 7f83cc612640 1 -- 172.21.15.119:0/1529274305 shutdown_connections 2024-09-10T19:02:51.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.875+0000 7f83cc612640 1 --2- 172.21.15.119:0/1529274305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83c4106e70 0x7f83c4109250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:51.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.876+0000 7f83cc612640 1 --2- 172.21.15.119:0/1529274305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83c40fe260 0x7f83c4106930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:51.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.876+0000 7f83cc612640 1 --2- 172.21.15.119:0/1529274305 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83c40fd8c0 0x7f83c40fdc90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:51.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.876+0000 7f83cc612640 1 -- 172.21.15.119:0/1529274305 >> 172.21.15.119:0/1529274305 conn(0x7f83c4071c00 msgr2=0x7f83c4072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:51.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.876+0000 7f83cc612640 1 -- 172.21.15.119:0/1529274305 shutdown_connections 2024-09-10T19:02:51.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.876+0000 7f83cc612640 1 -- 172.21.15.119:0/1529274305 wait complete. 2024-09-10T19:02:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.876+0000 7f83cc612640 1 Processor -- start 2024-09-10T19:02:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.876+0000 7f83cc612640 1 -- start start 2024-09-10T19:02:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.876+0000 7f83cc612640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83c40fd8c0 0x7f83c40fb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83cc612640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83c40fe260 0x7f83c40fbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83cc612640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83c4106e70 0x7f83c40fc200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83cc612640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c411b410 con 0x7f83c40fe260 2024-09-10T19:02:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83cc612640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83c411b580 con 0x7f83c40fd8c0 2024-09-10T19:02:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83cc612640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83c411b740 con 0x7f83c4106e70 2024-09-10T19:02:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83c9bad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83c40fe260 0x7f83c40fbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83cabaf640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83c4106e70 0x7f83c40fc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83c9bad640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83c40fe260 0x7f83c40fbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34758/0 (socket says 172.21.15.119:34758) 2024-09-10T19:02:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83cabaf640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83c4106e70 0x7f83c40fc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:48834/0 (socket says 172.21.15.119:48834) 2024-09-10T19:02:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83ca3ae640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83c40fd8c0 0x7f83c40fb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:51.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83c9bad640 1 -- 172.21.15.119:0/951120231 learned_addr learned my addr 172.21.15.119:0/951120231 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83c9bad640 1 -- 172.21.15.119:0/951120231 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83c4106e70 msgr2=0x7f83c40fc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83c9bad640 1 --2- 172.21.15.119:0/951120231 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83c4106e70 0x7f83c40fc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83c9bad640 1 -- 172.21.15.119:0/951120231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83c40fd8c0 msgr2=0x7f83c40fb780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83c9bad640 1 --2- 172.21.15.119:0/951120231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83c40fd8c0 0x7f83c40fb780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83c9bad640 1 -- 172.21.15.119:0/951120231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c00098b0 con 0x7f83c40fe260 2024-09-10T19:02:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83cabaf640 1 --2- 172.21.15.119:0/951120231 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83c4106e70 0x7f83c40fc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:02:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83c9bad640 1 --2- 172.21.15.119:0/951120231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83c40fe260 0x7f83c40fbcc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f83b400ebf0 tx=0x7f83b400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83baffd640 1 -- 172.21.15.119:0/951120231 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b4007c30 con 0x7f83c40fe260 2024-09-10T19:02:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83cc612640 1 -- 172.21.15.119:0/951120231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83bc005040 con 0x7f83c40fe260 2024-09-10T19:02:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83baffd640 1 -- 172.21.15.119:0/951120231 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f83b400f040 con 0x7f83c40fe260 2024-09-10T19:02:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83baffd640 1 -- 172.21.15.119:0/951120231 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83b40094f0 con 0x7f83c40fe260 2024-09-10T19:02:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.877+0000 7f83cc612640 1 -- 172.21.15.119:0/951120231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c411bd50 con 0x7f83c40fe260 2024-09-10T19:02:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.878+0000 7f83cc612640 1 -- 172.21.15.119:0/951120231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c4066310 con 0x7f83c40fe260 2024-09-10T19:02:51.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.879+0000 7f83baffd640 1 -- 172.21.15.119:0/951120231 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f83b4009860 con 0x7f83c40fe260 2024-09-10T19:02:51.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.879+0000 7f83baffd640 1 --2- 172.21.15.119:0/951120231 >> v2:172.21.15.119:6800/3118961733 conn(0x7f83b006eed0 0x7f83b0071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:51.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.879+0000 7f83ca3ae640 1 --2- 172.21.15.119:0/951120231 >> v2:172.21.15.119:6800/3118961733 conn(0x7f83b006eed0 0x7f83b0071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:51.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.879+0000 7f83baffd640 1 -- 172.21.15.119:0/951120231 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f83b408e1a0 con 0x7f83c40fe260 2024-09-10T19:02:51.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.880+0000 7f83ca3ae640 1 --2- 172.21.15.119:0/951120231 >> v2:172.21.15.119:6800/3118961733 conn(0x7f83b006eed0 0x7f83b0071380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f83bc009040 tx=0x7f83bc009da0 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:51.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.884+0000 7f83baffd640 1 -- 172.21.15.119:0/951120231 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f83b404e2a0 con 0x7f83c40fe260 2024-09-10T19:02:52.038 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:51 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:02:51] ENGINE Bus STOPPING 2024-09-10T19:02:52.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:52.045+0000 7f83cc612640 1 -- 172.21.15.119:0/951120231 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f83c40fcc60 con 0x7f83c40fe260 2024-09-10T19:02:52.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:52.045+0000 7f83baffd640 1 -- 172.21.15.119:0/951120231 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f83b405b170 con 0x7f83c40fe260 2024-09-10T19:02:52.047 INFO:teuthology.orchestra.run.smithi119.stdout:154618822684 2024-09-10T19:02:52.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:52.047+0000 7f83cc612640 1 -- 172.21.15.119:0/951120231 >> v2:172.21.15.119:6800/3118961733 conn(0x7f83b006eed0 msgr2=0x7f83b0071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:52.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:52.047+0000 7f83cc612640 1 --2- 172.21.15.119:0/951120231 >> v2:172.21.15.119:6800/3118961733 conn(0x7f83b006eed0 0x7f83b0071380 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f83bc009040 tx=0x7f83bc009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:52.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:52.047+0000 7f83cc612640 1 -- 172.21.15.119:0/951120231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83c40fe260 msgr2=0x7f83c40fbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:52.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:52.047+0000 7f83cc612640 1 --2- 172.21.15.119:0/951120231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83c40fe260 0x7f83c40fbcc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f83b400ebf0 tx=0x7f83b400b040 comp rx=0 tx=0).stop 2024-09-10T19:02:52.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:52.047+0000 7f83cc612640 1 -- 172.21.15.119:0/951120231 shutdown_connections 2024-09-10T19:02:52.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:52.047+0000 7f83cc612640 1 --2- 172.21.15.119:0/951120231 >> v2:172.21.15.119:6800/3118961733 conn(0x7f83b006eed0 0x7f83b0071380 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:52.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:52.047+0000 7f83cc612640 1 --2- 172.21.15.119:0/951120231 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f83c4106e70 0x7f83c40fc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:52.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:52.047+0000 7f83cc612640 1 --2- 172.21.15.119:0/951120231 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83c40fe260 0x7f83c40fbcc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:52.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:52.047+0000 7f83cc612640 1 --2- 172.21.15.119:0/951120231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83c40fd8c0 0x7f83c40fb780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:52.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:52.047+0000 7f83cc612640 1 -- 172.21.15.119:0/951120231 >> 172.21.15.119:0/951120231 conn(0x7f83c4071c00 msgr2=0x7f83c4107540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:52.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:52.047+0000 7f83cc612640 1 -- 172.21.15.119:0/951120231 shutdown_connections 2024-09-10T19:02:52.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:52.047+0000 7f83cc612640 1 -- 172.21.15.119:0/951120231 wait complete. 2024-09-10T19:02:52.316 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:02:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T19:02:52.317 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:02:52] ENGINE Bus STOPPED 2024-09-10T19:02:52.317 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:02:52] ENGINE Bus STARTING 2024-09-10T19:02:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[29577]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1297328730' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T19:02:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T19:02:52.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/951120231' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T19:02:52.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[40674]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1297328730' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T19:02:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T19:02:52.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/951120231' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T19:02:52.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:52 smithi194 ceph-mon[32464]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:52.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1297328730' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T19:02:52.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T19:02:52.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/951120231' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T19:02:52.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:02:52.595 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:02:52] ENGINE Serving on http://:::9283 2024-09-10T19:02:52.596 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:02:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:02:52] ENGINE Bus STARTED 2024-09-10T19:02:53.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:53 smithi119 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T19:02:53.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:53 smithi119 ceph-mon[29577]: Deploying daemon grafana.a on smithi194 2024-09-10T19:02:53.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:53 smithi119 ceph-mon[40674]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T19:02:53.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:53 smithi119 ceph-mon[40674]: Deploying daemon grafana.a on smithi194 2024-09-10T19:02:53.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:53 smithi194 ceph-mon[32464]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T19:02:53.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:53 smithi194 ceph-mon[32464]: Deploying daemon grafana.a on smithi194 2024-09-10T19:02:53.615 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:02:53 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[75363]: ts=2024-09-10T19:02:53.571Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000167654s 2024-09-10T19:02:54.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.526+0000 7f8efdaa7640 1 -- 172.21.15.119:0/2635854481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ef80fb800 msgr2=0x7f8ef80fbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.526+0000 7f8efdaa7640 1 --2- 172.21.15.119:0/2635854481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ef80fb800 0x7f8ef80fbc70 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0004980 tx=0x7f8ee0009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.526+0000 7f8efdaa7640 1 -- 172.21.15.119:0/2635854481 shutdown_connections 2024-09-10T19:02:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.526+0000 7f8efdaa7640 1 --2- 172.21.15.119:0/2635854481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80fc1b0 0x7f8ef8108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.526+0000 7f8efdaa7640 1 --2- 172.21.15.119:0/2635854481 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ef80fb800 0x7f8ef80fbc70 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.526+0000 7f8efdaa7640 1 --2- 172.21.15.119:0/2635854481 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8ef81017d0 0x7f8ef8101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.526+0000 7f8efdaa7640 1 -- 172.21.15.119:0/2635854481 >> 172.21.15.119:0/2635854481 conn(0x7f8ef8071c00 msgr2=0x7f8ef8072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.526+0000 7f8efdaa7640 1 -- 172.21.15.119:0/2635854481 shutdown_connections 2024-09-10T19:02:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.526+0000 7f8efdaa7640 1 -- 172.21.15.119:0/2635854481 wait complete. 2024-09-10T19:02:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.526+0000 7f8efdaa7640 1 Processor -- start 2024-09-10T19:02:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.526+0000 7f8efdaa7640 1 -- start start 2024-09-10T19:02:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8efdaa7640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80fb800 0x7f8ef8111c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8efdaa7640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8ef80fc1b0 0x7f8ef81121d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8efdaa7640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ef81017d0 0x7f8ef810bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8efdaa7640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef810c4a0 con 0x7f8ef81017d0 2024-09-10T19:02:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8efdaa7640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef810c610 con 0x7f8ef80fb800 2024-09-10T19:02:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8efdaa7640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef810c7d0 con 0x7f8ef80fc1b0 2024-09-10T19:02:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8eeffff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8ef80fc1b0 0x7f8ef81121d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8ef7fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ef81017d0 0x7f8ef810bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8eeffff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8ef80fc1b0 0x7f8ef81121d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:48808/0 (socket says 172.21.15.119:48808) 2024-09-10T19:02:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8ef7fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ef81017d0 0x7f8ef810bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34748/0 (socket says 172.21.15.119:34748) 2024-09-10T19:02:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8eeffff640 1 -- 172.21.15.119:0/1297328730 learned_addr learned my addr 172.21.15.119:0/1297328730 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8ef77fe640 1 --2- 172.21.15.119:0/1297328730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80fb800 0x7f8ef8111c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8eeffff640 1 -- 172.21.15.119:0/1297328730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80fb800 msgr2=0x7f8ef8111c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8eeffff640 1 --2- 172.21.15.119:0/1297328730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80fb800 0x7f8ef8111c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8eeffff640 1 -- 172.21.15.119:0/1297328730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ef81017d0 msgr2=0x7f8ef810bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8eeffff640 1 --2- 172.21.15.119:0/1297328730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ef81017d0 0x7f8ef810bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8eeffff640 1 -- 172.21.15.119:0/1297328730 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ee8005040 con 0x7f8ef80fc1b0 2024-09-10T19:02:54.005 INFO:teuthology.orchestra.run.smithi119.stdout:55834574896 2024-09-10T19:02:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8ef7fff640 1 --2- 172.21.15.119:0/1297328730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ef81017d0 0x7f8ef810bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:02:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8eeffff640 1 --2- 172.21.15.119:0/1297328730 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8ef80fc1b0 0x7f8ef81121d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8ee00094b0 tx=0x7f8ee0009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:54.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8ef4ff9640 1 -- 172.21.15.119:0/1297328730 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee002bcb0 con 0x7f8ef80fc1b0 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8ef4ff9640 1 -- 172.21.15.119:0/1297328730 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f8ee002be10 con 0x7f8ef80fc1b0 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8efdaa7640 1 -- 172.21.15.119:0/1297328730 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ee0005040 con 0x7f8ef80fc1b0 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8ef4ff9640 1 -- 172.21.15.119:0/1297328730 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0033420 con 0x7f8ef80fc1b0 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.527+0000 7f8efdaa7640 1 -- 172.21.15.119:0/1297328730 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef8065250 con 0x7f8ef80fc1b0 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.528+0000 7f8efdaa7640 1 -- 172.21.15.119:0/1297328730 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef80fcef0 con 0x7f8ef80fc1b0 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.529+0000 7f8ef4ff9640 1 -- 172.21.15.119:0/1297328730 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f8ee0030020 con 0x7f8ef80fc1b0 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.529+0000 7f8ef4ff9640 1 --2- 172.21.15.119:0/1297328730 >> v2:172.21.15.119:6800/3118961733 conn(0x7f8ee406eed0 0x7f8ee4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.529+0000 7f8ef77fe640 1 --2- 172.21.15.119:0/1297328730 >> v2:172.21.15.119:6800/3118961733 conn(0x7f8ee406eed0 0x7f8ee4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.529+0000 7f8ef4ff9640 1 -- 172.21.15.119:0/1297328730 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f8ee00ae0a0 con 0x7f8ef80fc1b0 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.530+0000 7f8ef77fe640 1 --2- 172.21.15.119:0/1297328730 >> v2:172.21.15.119:6800/3118961733 conn(0x7f8ee406eed0 0x7f8ee4071380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8004a50 tx=0x7f8ee800b040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.534+0000 7f8ef4ff9640 1 -- 172.21.15.119:0/1297328730 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f8ee007b030 con 0x7f8ef80fc1b0 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.712+0000 7f8efdaa7640 1 -- 172.21.15.119:0/1297328730 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8ef810d3b0 con 0x7f8ef80fc1b0 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.713+0000 7f8ef4ff9640 1 -- 172.21.15.119:0/1297328730 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8ee007a9d0 con 0x7f8ef80fc1b0 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.715+0000 7f8efdaa7640 1 -- 172.21.15.119:0/1297328730 >> v2:172.21.15.119:6800/3118961733 conn(0x7f8ee406eed0 msgr2=0x7f8ee4071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.715+0000 7f8efdaa7640 1 --2- 172.21.15.119:0/1297328730 >> v2:172.21.15.119:6800/3118961733 conn(0x7f8ee406eed0 0x7f8ee4071380 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8004a50 tx=0x7f8ee800b040 comp rx=0 tx=0).stop 2024-09-10T19:02:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.715+0000 7f8efdaa7640 1 -- 172.21.15.119:0/1297328730 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8ef80fc1b0 msgr2=0x7f8ef81121d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.715+0000 7f8efdaa7640 1 --2- 172.21.15.119:0/1297328730 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8ef80fc1b0 0x7f8ef81121d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8ee00094b0 tx=0x7f8ee0009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.715+0000 7f8efdaa7640 1 -- 172.21.15.119:0/1297328730 shutdown_connections 2024-09-10T19:02:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.715+0000 7f8efdaa7640 1 --2- 172.21.15.119:0/1297328730 >> v2:172.21.15.119:6800/3118961733 conn(0x7f8ee406eed0 0x7f8ee4071380 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.715+0000 7f8efdaa7640 1 --2- 172.21.15.119:0/1297328730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ef81017d0 0x7f8ef810bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.715+0000 7f8efdaa7640 1 --2- 172.21.15.119:0/1297328730 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8ef80fc1b0 0x7f8ef81121d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.715+0000 7f8efdaa7640 1 --2- 172.21.15.119:0/1297328730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef80fb800 0x7f8ef8111c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.715+0000 7f8efdaa7640 1 -- 172.21.15.119:0/1297328730 >> 172.21.15.119:0/1297328730 conn(0x7f8ef8071c00 msgr2=0x7f8ef80fa2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.715+0000 7f8efdaa7640 1 -- 172.21.15.119:0/1297328730 shutdown_connections 2024-09-10T19:02:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:51.715+0000 7f8efdaa7640 1 -- 172.21.15.119:0/1297328730 wait complete. 2024-09-10T19:02:54.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:54 smithi194 ceph-mon[32464]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:54.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:54 smithi119 ceph-mon[29577]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:54.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:54 smithi119 ceph-mon[40674]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:02:55.274 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/577316877' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T19:02:55.275 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/577316877' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T19:02:55.276 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574896 for osd.1 2024-09-10T19:02:55.276 DEBUG:teuthology.parallel:result is None 2024-09-10T19:02:55.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/577316877' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T19:02:55.940 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822681 got 154618822684 for osd.5 2024-09-10T19:02:55.941 DEBUG:teuthology.parallel:result is None 2024-09-10T19:02:56.310 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:56 smithi119 ceph-mon[29577]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:56.310 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:56 smithi119 ceph-mon[40674]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:56.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:56 smithi194 ceph-mon[32464]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:56.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.759+0000 7f2950616640 1 -- 172.21.15.119:0/1117360179 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f29481028c0 msgr2=0x7f2948102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:56.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.759+0000 7f2950616640 1 --2- 172.21.15.119:0/1117360179 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f29481028c0 0x7f2948102c90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2938004980 tx=0x7f2938009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:56.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.759+0000 7f2950616640 1 -- 172.21.15.119:0/1117360179 shutdown_connections 2024-09-10T19:02:56.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.759+0000 7f2950616640 1 --2- 172.21.15.119:0/1117360179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29480fd250 0x7f2948109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:56.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.759+0000 7f2950616640 1 --2- 172.21.15.119:0/1117360179 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29480fc8a0 0x7f29480fcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:56.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.759+0000 7f2950616640 1 --2- 172.21.15.119:0/1117360179 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f29481028c0 0x7f2948102c90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:56.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.759+0000 7f2950616640 1 -- 172.21.15.119:0/1117360179 >> 172.21.15.119:0/1117360179 conn(0x7f29480f8400 msgr2=0x7f29480fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:56.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.759+0000 7f2950616640 1 -- 172.21.15.119:0/1117360179 shutdown_connections 2024-09-10T19:02:56.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.759+0000 7f2950616640 1 -- 172.21.15.119:0/1117360179 wait complete. 2024-09-10T19:02:56.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.759+0000 7f2950616640 1 Processor -- start 2024-09-10T19:02:56.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.759+0000 7f2950616640 1 -- start start 2024-09-10T19:02:56.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f2950616640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29480fc8a0 0x7f2948112dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:56.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f2950616640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f29480fd250 0x7f2948113310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:56.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f2950616640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29481028c0 0x7f294810cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:56.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f2950616640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294810d5b0 con 0x7f29480fc8a0 2024-09-10T19:02:56.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f2950616640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294810d720 con 0x7f29481028c0 2024-09-10T19:02:56.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f2950616640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294810d8e0 con 0x7f29480fd250 2024-09-10T19:02:56.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f294e3b2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29480fc8a0 0x7f2948112dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:56.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f294e3b2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29480fc8a0 0x7f2948112dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40052/0 (socket says 172.21.15.119:40052) 2024-09-10T19:02:56.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f294e3b2640 1 -- 172.21.15.119:0/2707433787 learned_addr learned my addr 172.21.15.119:0/2707433787 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:56.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f294dbb1640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f29480fd250 0x7f2948113310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f294ebb3640 1 --2- 172.21.15.119:0/2707433787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29481028c0 0x7f294810cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f294e3b2640 1 -- 172.21.15.119:0/2707433787 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f29480fd250 msgr2=0x7f2948113310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f294e3b2640 1 --2- 172.21.15.119:0/2707433787 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f29480fd250 0x7f2948113310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f294e3b2640 1 -- 172.21.15.119:0/2707433787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29481028c0 msgr2=0x7f294810cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f294e3b2640 1 --2- 172.21.15.119:0/2707433787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29481028c0 0x7f294810cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f294e3b2640 1 -- 172.21.15.119:0/2707433787 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29400098b0 con 0x7f29480fc8a0 2024-09-10T19:02:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f294dbb1640 1 --2- 172.21.15.119:0/2707433787 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f29480fd250 0x7f2948113310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:02:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f294e3b2640 1 --2- 172.21.15.119:0/2707433787 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29480fc8a0 0x7f2948112dd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f29380094b0 tx=0x7f2938009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f293effd640 1 -- 172.21.15.119:0/2707433787 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293802bcb0 con 0x7f29480fc8a0 2024-09-10T19:02:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f293effd640 1 -- 172.21.15.119:0/2707433787 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f293802be10 con 0x7f29480fc8a0 2024-09-10T19:02:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f2950616640 1 -- 172.21.15.119:0/2707433787 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2938005040 con 0x7f29480fc8a0 2024-09-10T19:02:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f293effd640 1 -- 172.21.15.119:0/2707433787 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29380336b0 con 0x7f29480fc8a0 2024-09-10T19:02:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.760+0000 7f2950616640 1 -- 172.21.15.119:0/2707433787 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294806f940 con 0x7f29480fc8a0 2024-09-10T19:02:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.761+0000 7f2950616640 1 -- 172.21.15.119:0/2707433787 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29480fdfe0 con 0x7f29480fc8a0 2024-09-10T19:02:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.762+0000 7f293effd640 1 -- 172.21.15.119:0/2707433787 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f2938030020 con 0x7f29480fc8a0 2024-09-10T19:02:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.762+0000 7f293effd640 1 --2- 172.21.15.119:0/2707433787 >> v2:172.21.15.119:6800/3118961733 conn(0x7f293406eed0 0x7f2934071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.762+0000 7f293effd640 1 -- 172.21.15.119:0/2707433787 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f29380ae080 con 0x7f29480fc8a0 2024-09-10T19:02:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.762+0000 7f294dbb1640 1 --2- 172.21.15.119:0/2707433787 >> v2:172.21.15.119:6800/3118961733 conn(0x7f293406eed0 0x7f2934071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.763+0000 7f294dbb1640 1 --2- 172.21.15.119:0/2707433787 >> v2:172.21.15.119:6800/3118961733 conn(0x7f293406eed0 0x7f2934071380 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2944004fd0 tx=0x7f2944011040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.766+0000 7f293effd640 1 -- 172.21.15.119:0/2707433787 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f293807b0d0 con 0x7f29480fc8a0 2024-09-10T19:02:56.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.930+0000 7f2950616640 1 -- 172.21.15.119:0/2707433787 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f294810bd60 con 0x7f29480fc8a0 2024-09-10T19:02:56.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.931+0000 7f293effd640 1 -- 172.21.15.119:0/2707433787 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f293807aa70 con 0x7f29480fc8a0 2024-09-10T19:02:56.932 INFO:teuthology.orchestra.run.smithi119.stdout:77309411371 2024-09-10T19:02:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.932+0000 7f2950616640 1 -- 172.21.15.119:0/2707433787 >> v2:172.21.15.119:6800/3118961733 conn(0x7f293406eed0 msgr2=0x7f2934071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.932+0000 7f2950616640 1 --2- 172.21.15.119:0/2707433787 >> v2:172.21.15.119:6800/3118961733 conn(0x7f293406eed0 0x7f2934071380 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2944004fd0 tx=0x7f2944011040 comp rx=0 tx=0).stop 2024-09-10T19:02:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.933+0000 7f2950616640 1 -- 172.21.15.119:0/2707433787 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29480fc8a0 msgr2=0x7f2948112dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.933+0000 7f2950616640 1 --2- 172.21.15.119:0/2707433787 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29480fc8a0 0x7f2948112dd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f29380094b0 tx=0x7f2938009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.933+0000 7f2950616640 1 -- 172.21.15.119:0/2707433787 shutdown_connections 2024-09-10T19:02:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.933+0000 7f2950616640 1 --2- 172.21.15.119:0/2707433787 >> v2:172.21.15.119:6800/3118961733 conn(0x7f293406eed0 0x7f2934071380 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.933+0000 7f2950616640 1 --2- 172.21.15.119:0/2707433787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29481028c0 0x7f294810cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.933+0000 7f2950616640 1 --2- 172.21.15.119:0/2707433787 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f29480fd250 0x7f2948113310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.933+0000 7f2950616640 1 --2- 172.21.15.119:0/2707433787 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29480fc8a0 0x7f2948112dd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.933+0000 7f2950616640 1 -- 172.21.15.119:0/2707433787 >> 172.21.15.119:0/2707433787 conn(0x7f29480f8400 msgr2=0x7f29480f9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:56.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.933+0000 7f2950616640 1 -- 172.21.15.119:0/2707433787 shutdown_connections 2024-09-10T19:02:56.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:56.933+0000 7f2950616640 1 -- 172.21.15.119:0/2707433787 wait complete. 2024-09-10T19:02:57.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2707433787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T19:02:57.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2707433787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T19:02:57.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2707433787' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T19:02:57.735 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411371 for osd.2 2024-09-10T19:02:57.735 DEBUG:teuthology.parallel:result is None 2024-09-10T19:02:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.899+0000 7fc1e5758640 1 -- 172.21.15.119:0/279758124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1e01017f0 msgr2=0x7fc1e0101bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:57.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.899+0000 7fc1e5758640 1 --2- 172.21.15.119:0/279758124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1e01017f0 0x7fc1e0101bc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0004890 tx=0x7fc1d0009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.899+0000 7fc1e5758640 1 -- 172.21.15.119:0/279758124 shutdown_connections 2024-09-10T19:02:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.899+0000 7fc1e5758640 1 --2- 172.21.15.119:0/279758124 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1e00fc630 0x7fc1e01085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.899+0000 7fc1e5758640 1 --2- 172.21.15.119:0/279758124 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc1e0102190 0x7fc1e00fbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.899+0000 7fc1e5758640 1 --2- 172.21.15.119:0/279758124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1e01017f0 0x7fc1e0101bc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.899+0000 7fc1e5758640 1 -- 172.21.15.119:0/279758124 >> 172.21.15.119:0/279758124 conn(0x7fc1e0074d40 msgr2=0x7fc1e00f9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.899+0000 7fc1e5758640 1 -- 172.21.15.119:0/279758124 shutdown_connections 2024-09-10T19:02:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.899+0000 7fc1e5758640 1 -- 172.21.15.119:0/279758124 wait complete. 2024-09-10T19:02:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1e5758640 1 Processor -- start 2024-09-10T19:02:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1e5758640 1 -- start start 2024-09-10T19:02:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1e5758640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1e00fc630 0x7fc1e0111d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1e5758640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1e01017f0 0x7fc1e0112280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:57.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1e5758640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc1e0102190 0x7fc1e010bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:57.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1e5758640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e010c520 con 0x7fc1e01017f0 2024-09-10T19:02:57.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1e5758640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e010c690 con 0x7fc1e0102190 2024-09-10T19:02:57.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1e5758640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e010c850 con 0x7fc1e00fc630 2024-09-10T19:02:57.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1deffd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1e00fc630 0x7fc1e0111d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:57.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1d7fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1e01017f0 0x7fc1e0112280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:57.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1d7fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1e01017f0 0x7fc1e0112280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40074/0 (socket says 172.21.15.119:40074) 2024-09-10T19:02:57.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1d7fff640 1 -- 172.21.15.119:0/1430423263 learned_addr learned my addr 172.21.15.119:0/1430423263 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:02:57.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1deffd640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1e00fc630 0x7fc1e0111d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:50440/0 (socket says 172.21.15.119:50440) 2024-09-10T19:02:57.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1df7fe640 1 --2- 172.21.15.119:0/1430423263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc1e0102190 0x7fc1e010bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:57.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1d7fff640 1 -- 172.21.15.119:0/1430423263 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1e00fc630 msgr2=0x7fc1e0111d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:57.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1d7fff640 1 --2- 172.21.15.119:0/1430423263 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1e00fc630 0x7fc1e0111d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:57.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1d7fff640 1 -- 172.21.15.119:0/1430423263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc1e0102190 msgr2=0x7fc1e010bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:57.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1d7fff640 1 --2- 172.21.15.119:0/1430423263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc1e0102190 0x7fc1e010bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:57.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1d7fff640 1 -- 172.21.15.119:0/1430423263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d80098b0 con 0x7fc1e01017f0 2024-09-10T19:02:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1deffd640 1 --2- 172.21.15.119:0/1430423263 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1e00fc630 0x7fc1e0111d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:02:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.900+0000 7fc1d7fff640 1 --2- 172.21.15.119:0/1430423263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1e01017f0 0x7fc1e0112280 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc1c80044e0 tx=0x7fc1c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.901+0000 7fc1d77fe640 1 -- 172.21.15.119:0/1430423263 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8007c30 con 0x7fc1e01017f0 2024-09-10T19:02:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.901+0000 7fc1d77fe640 1 -- 172.21.15.119:0/1430423263 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fc1c800f040 con 0x7fc1e01017f0 2024-09-10T19:02:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.901+0000 7fc1d77fe640 1 -- 172.21.15.119:0/1430423263 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c8009340 con 0x7fc1e01017f0 2024-09-10T19:02:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.901+0000 7fc1e5758640 1 -- 172.21.15.119:0/1430423263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d0005040 con 0x7fc1e01017f0 2024-09-10T19:02:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.901+0000 7fc1e5758640 1 -- 172.21.15.119:0/1430423263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e0118b90 con 0x7fc1e01017f0 2024-09-10T19:02:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.902+0000 7fc1e5758640 1 -- 172.21.15.119:0/1430423263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e00fd120 con 0x7fc1e01017f0 2024-09-10T19:02:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.902+0000 7fc1d77fe640 1 -- 172.21.15.119:0/1430423263 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7fc1c8009770 con 0x7fc1e01017f0 2024-09-10T19:02:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.902+0000 7fc1d77fe640 1 --2- 172.21.15.119:0/1430423263 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc1cc06eed0 0x7fc1cc071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:02:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.903+0000 7fc1d77fe640 1 -- 172.21.15.119:0/1430423263 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7fc1c808e090 con 0x7fc1e01017f0 2024-09-10T19:02:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.903+0000 7fc1deffd640 1 --2- 172.21.15.119:0/1430423263 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc1cc06eed0 0x7fc1cc071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:02:57.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.903+0000 7fc1deffd640 1 --2- 172.21.15.119:0/1430423263 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc1cc06eed0 0x7fc1cc071380 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0009f80 tx=0x7fc1d0009da0 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:02:57.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:57.907+0000 7fc1d77fe640 1 -- 172.21.15.119:0/1430423263 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fc1c804e2a0 con 0x7fc1e01017f0 2024-09-10T19:02:58.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:58.069+0000 7fc1e5758640 1 -- 172.21.15.119:0/1430423263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc1e010d3e0 con 0x7fc1e01017f0 2024-09-10T19:02:58.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:58.069+0000 7fc1d77fe640 1 -- 172.21.15.119:0/1430423263 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc1c805b020 con 0x7fc1e01017f0 2024-09-10T19:02:58.071 INFO:teuthology.orchestra.run.smithi119.stdout:180388626458 2024-09-10T19:02:58.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:58.071+0000 7fc1e5758640 1 -- 172.21.15.119:0/1430423263 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc1cc06eed0 msgr2=0x7fc1cc071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:58.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:58.071+0000 7fc1e5758640 1 --2- 172.21.15.119:0/1430423263 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc1cc06eed0 0x7fc1cc071380 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc1d0009f80 tx=0x7fc1d0009da0 comp rx=0 tx=0).stop 2024-09-10T19:02:58.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:58.071+0000 7fc1e5758640 1 -- 172.21.15.119:0/1430423263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1e01017f0 msgr2=0x7fc1e0112280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:02:58.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:58.071+0000 7fc1e5758640 1 --2- 172.21.15.119:0/1430423263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1e01017f0 0x7fc1e0112280 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc1c80044e0 tx=0x7fc1c800b040 comp rx=0 tx=0).stop 2024-09-10T19:02:58.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:58.071+0000 7fc1e5758640 1 -- 172.21.15.119:0/1430423263 shutdown_connections 2024-09-10T19:02:58.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:58.071+0000 7fc1e5758640 1 --2- 172.21.15.119:0/1430423263 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc1cc06eed0 0x7fc1cc071380 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:58.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:58.071+0000 7fc1e5758640 1 --2- 172.21.15.119:0/1430423263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc1e0102190 0x7fc1e010bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:58.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:58.071+0000 7fc1e5758640 1 --2- 172.21.15.119:0/1430423263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1e01017f0 0x7fc1e0112280 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:58.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:58.071+0000 7fc1e5758640 1 --2- 172.21.15.119:0/1430423263 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc1e00fc630 0x7fc1e0111d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:02:58.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:58.071+0000 7fc1e5758640 1 -- 172.21.15.119:0/1430423263 >> 172.21.15.119:0/1430423263 conn(0x7fc1e0074d40 msgr2=0x7fc1e0071f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:02:58.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:58.071+0000 7fc1e5758640 1 -- 172.21.15.119:0/1430423263 shutdown_connections 2024-09-10T19:02:58.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:58.071+0000 7fc1e5758640 1 -- 172.21.15.119:0/1430423263 wait complete. 2024-09-10T19:02:58.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:58 smithi119 ceph-mon[29577]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:58.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1430423263' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T19:02:58.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:58 smithi119 ceph-mon[40674]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:58.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1430423263' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T19:02:58.419 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:58 smithi194 ceph-mon[32464]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:02:58.419 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1430423263' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T19:02:58.420 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:02:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:02:59.144 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626454 got 180388626458 for osd.6 2024-09-10T19:02:59.144 DEBUG:teuthology.parallel:result is None 2024-09-10T19:02:59.418 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:02:59 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:59.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:02:59 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:02:59.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:02:59 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:00.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.070+0000 7fa18dce0640 1 -- 172.21.15.119:0/262658817 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1880fb810 msgr2=0x7fa1880fbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:00.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.070+0000 7fa18dce0640 1 --2- 172.21.15.119:0/262658817 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1880fb810 0x7fa1880fbc80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa17c004980 tx=0x7fa17c009da0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.070+0000 7fa18dce0640 1 -- 172.21.15.119:0/262658817 shutdown_connections 2024-09-10T19:03:00.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.070+0000 7fa18dce0640 1 --2- 172.21.15.119:0/262658817 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1880fc1c0 0x7fa1881085e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.070+0000 7fa18dce0640 1 --2- 172.21.15.119:0/262658817 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1880fb810 0x7fa1880fbc80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.070+0000 7fa18dce0640 1 --2- 172.21.15.119:0/262658817 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa188101830 0x7fa188101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.070+0000 7fa18dce0640 1 -- 172.21.15.119:0/262658817 >> 172.21.15.119:0/262658817 conn(0x7fa1880713f0 msgr2=0x7fa1880f99f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:00.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.070+0000 7fa18dce0640 1 -- 172.21.15.119:0/262658817 shutdown_connections 2024-09-10T19:03:00.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.070+0000 7fa18dce0640 1 -- 172.21.15.119:0/262658817 wait complete. 2024-09-10T19:03:00.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.070+0000 7fa18dce0640 1 Processor -- start 2024-09-10T19:03:00.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.070+0000 7fa18dce0640 1 -- start start 2024-09-10T19:03:00.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa18dce0640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1880fb810 0x7fa188111cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:00.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa18dce0640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1880fc1c0 0x7fa188112200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:00.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa18dce0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa188101830 0x7fa18810be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:00.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa18dce0640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18810c460 con 0x7fa188101830 2024-09-10T19:03:00.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa18dce0640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18810c5d0 con 0x7fa1880fc1c0 2024-09-10T19:03:00.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa18dce0640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18810c790 con 0x7fa1880fb810 2024-09-10T19:03:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa1877fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1880fb810 0x7fa188111cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa1877fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1880fb810 0x7fa188111cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:50460/0 (socket says 172.21.15.119:50460) 2024-09-10T19:03:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa1877fe640 1 -- 172.21.15.119:0/2982993139 learned_addr learned my addr 172.21.15.119:0/2982993139 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:03:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa187fff640 1 --2- 172.21.15.119:0/2982993139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa188101830 0x7fa18810be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa186ffd640 1 --2- 172.21.15.119:0/2982993139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1880fc1c0 0x7fa188112200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa1877fe640 1 -- 172.21.15.119:0/2982993139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1880fc1c0 msgr2=0x7fa188112200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa1877fe640 1 --2- 172.21.15.119:0/2982993139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1880fc1c0 0x7fa188112200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa1877fe640 1 -- 172.21.15.119:0/2982993139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa188101830 msgr2=0x7fa18810be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa1877fe640 1 --2- 172.21.15.119:0/2982993139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa188101830 0x7fa18810be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa1877fe640 1 -- 172.21.15.119:0/2982993139 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c005040 con 0x7fa1880fb810 2024-09-10T19:03:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa187fff640 1 --2- 172.21.15.119:0/2982993139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa188101830 0x7fa18810be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:03:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa1877fe640 1 --2- 172.21.15.119:0/2982993139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1880fb810 0x7fa188111cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa178010e00 tx=0x7fa17800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:03:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa16ffff640 1 -- 172.21.15.119:0/2982993139 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17800cb30 con 0x7fa1880fb810 2024-09-10T19:03:00.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa18dce0640 1 -- 172.21.15.119:0/2982993139 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa18810ca10 con 0x7fa1880fb810 2024-09-10T19:03:00.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa16ffff640 1 -- 172.21.15.119:0/2982993139 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fa178007e90 con 0x7fa1880fb810 2024-09-10T19:03:00.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.071+0000 7fa16ffff640 1 -- 172.21.15.119:0/2982993139 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1780095f0 con 0x7fa1880fb810 2024-09-10T19:03:00.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.072+0000 7fa18dce0640 1 -- 172.21.15.119:0/2982993139 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa188065fc0 con 0x7fa1880fb810 2024-09-10T19:03:00.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.072+0000 7fa18dce0640 1 -- 172.21.15.119:0/2982993139 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1880fcf50 con 0x7fa1880fb810 2024-09-10T19:03:00.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.073+0000 7fa16ffff640 1 -- 172.21.15.119:0/2982993139 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7fa178018030 con 0x7fa1880fb810 2024-09-10T19:03:00.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.073+0000 7fa16ffff640 1 --2- 172.21.15.119:0/2982993139 >> v2:172.21.15.119:6800/3118961733 conn(0x7fa17006eed0 0x7fa170071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:00.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.073+0000 7fa186ffd640 1 --2- 172.21.15.119:0/2982993139 >> v2:172.21.15.119:6800/3118961733 conn(0x7fa17006eed0 0x7fa170071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:00.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.073+0000 7fa16ffff640 1 -- 172.21.15.119:0/2982993139 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7fa178016070 con 0x7fa1880fb810 2024-09-10T19:03:00.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.074+0000 7fa186ffd640 1 --2- 172.21.15.119:0/2982993139 >> v2:172.21.15.119:6800/3118961733 conn(0x7fa17006eed0 0x7fa170071380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa17c009f80 tx=0x7fa17c009da0 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:03:00.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.077+0000 7fa16ffff640 1 -- 172.21.15.119:0/2982993139 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fa17805d6b0 con 0x7fa1880fb810 2024-09-10T19:03:00.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.847+0000 7f52df582640 1 -- 172.21.15.119:0/311208520 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52d80fb800 msgr2=0x7f52d80fbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:00.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.847+0000 7f52df582640 1 --2- 172.21.15.119:0/311208520 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52d80fb800 0x7f52d80fbc70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f52c80048c0 tx=0x7f52c8009da0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.847+0000 7f52df582640 1 -- 172.21.15.119:0/311208520 shutdown_connections 2024-09-10T19:03:00.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.847+0000 7f52df582640 1 --2- 172.21.15.119:0/311208520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52d80fc1b0 0x7f52d8108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.847+0000 7f52df582640 1 --2- 172.21.15.119:0/311208520 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52d80fb800 0x7f52d80fbc70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.847+0000 7f52df582640 1 --2- 172.21.15.119:0/311208520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52d81017d0 0x7f52d8101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.847+0000 7f52df582640 1 -- 172.21.15.119:0/311208520 >> 172.21.15.119:0/311208520 conn(0x7f52d8071c00 msgr2=0x7f52d8072000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:00.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.847+0000 7f52df582640 1 -- 172.21.15.119:0/311208520 shutdown_connections 2024-09-10T19:03:00.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.847+0000 7f52df582640 1 -- 172.21.15.119:0/311208520 wait complete. 2024-09-10T19:03:00.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52df582640 1 Processor -- start 2024-09-10T19:03:00.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52df582640 1 -- start start 2024-09-10T19:03:00.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52df582640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52d80fb800 0x7f52d810bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:00.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52df582640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52d80fc1b0 0x7f52d810c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:00.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52df582640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52d81017d0 0x7f52d810c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:00.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52df582640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d811b420 con 0x7f52d81017d0 2024-09-10T19:03:00.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52df582640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d811b560 con 0x7f52d80fb800 2024-09-10T19:03:00.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52df582640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52d811b6f0 con 0x7f52d80fc1b0 2024-09-10T19:03:00.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52dcb1d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52d80fc1b0 0x7f52d810c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:00.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52ddb1f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52d81017d0 0x7f52d810c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:00.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52dd31e640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52d80fb800 0x7f52d810bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:00.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52dcb1d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52d80fc1b0 0x7f52d810c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:50410/0 (socket says 172.21.15.119:50410) 2024-09-10T19:03:00.225 INFO:teuthology.orchestra.run.smithi119.stdout:206158430228 2024-09-10T19:03:00.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52dcb1d640 1 -- 172.21.15.119:0/577316877 learned_addr learned my addr 172.21.15.119:0/577316877 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:03:00.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52dd31e640 1 --2- 172.21.15.119:0/577316877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52d80fb800 0x7f52d810bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.119:46762/0 (socket says 172.21.15.119:46762) 2024-09-10T19:03:00.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52dcb1d640 1 -- 172.21.15.119:0/577316877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52d80fb800 msgr2=0x7f52d810bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:00.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52dcb1d640 1 --2- 172.21.15.119:0/577316877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52d80fb800 0x7f52d810bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.848+0000 7f52dcb1d640 1 -- 172.21.15.119:0/577316877 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52d81017d0 msgr2=0x7f52d810c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:00.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.849+0000 7f52dcb1d640 1 --2- 172.21.15.119:0/577316877 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52d81017d0 0x7f52d810c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.849+0000 7f52dcb1d640 1 -- 172.21.15.119:0/577316877 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d00098b0 con 0x7f52d80fc1b0 2024-09-10T19:03:00.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.849+0000 7f52dcb1d640 1 --2- 172.21.15.119:0/577316877 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52d80fc1b0 0x7f52d810c2e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f52c80040c0 tx=0x7f52c8009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:03:00.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.849+0000 7f52cdffb640 1 -- 172.21.15.119:0/577316877 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c80090d0 con 0x7f52d80fc1b0 2024-09-10T19:03:00.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.849+0000 7f52cdffb640 1 -- 172.21.15.119:0/577316877 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f52c8009230 con 0x7f52d80fc1b0 2024-09-10T19:03:00.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.849+0000 7f52df582640 1 -- 172.21.15.119:0/577316877 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52c8005040 con 0x7f52d80fc1b0 2024-09-10T19:03:00.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.849+0000 7f52cdffb640 1 -- 172.21.15.119:0/577316877 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8033690 con 0x7f52d80fc1b0 2024-09-10T19:03:00.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.849+0000 7f52df582640 1 -- 172.21.15.119:0/577316877 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d811bd30 con 0x7f52d80fc1b0 2024-09-10T19:03:00.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.850+0000 7f52df582640 1 -- 172.21.15.119:0/577316877 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d80fcef0 con 0x7f52d80fc1b0 2024-09-10T19:03:00.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.850+0000 7f52cdffb640 1 -- 172.21.15.119:0/577316877 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f52c8030020 con 0x7f52d80fc1b0 2024-09-10T19:03:00.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.851+0000 7f52cdffb640 1 --2- 172.21.15.119:0/577316877 >> v2:172.21.15.119:6800/3118961733 conn(0x7f52c406eed0 0x7f52c4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:00.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.851+0000 7f52cdffb640 1 -- 172.21.15.119:0/577316877 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f52c803ce40 con 0x7f52d80fc1b0 2024-09-10T19:03:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.851+0000 7f52dd31e640 1 --2- 172.21.15.119:0/577316877 >> v2:172.21.15.119:6800/3118961733 conn(0x7f52c406eed0 0x7f52c4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.851+0000 7f52dd31e640 1 --2- 172.21.15.119:0/577316877 >> v2:172.21.15.119:6800/3118961733 conn(0x7f52c406eed0 0x7f52c4071380 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f52d4009af0 tx=0x7f52d400b040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:03:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:54.855+0000 7f52cdffb640 1 -- 172.21.15.119:0/577316877 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f52c807b280 con 0x7f52d80fc1b0 2024-09-10T19:03:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:55.013+0000 7f52df582640 1 -- 172.21.15.119:0/577316877 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f52d80657a0 con 0x7f52d80fc1b0 2024-09-10T19:03:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:55.013+0000 7f52cdffb640 1 -- 172.21.15.119:0/577316877 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f52c807ac20 con 0x7f52d80fc1b0 2024-09-10T19:03:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:55.015+0000 7f52df582640 1 -- 172.21.15.119:0/577316877 >> v2:172.21.15.119:6800/3118961733 conn(0x7f52c406eed0 msgr2=0x7f52c4071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:55.015+0000 7f52df582640 1 --2- 172.21.15.119:0/577316877 >> v2:172.21.15.119:6800/3118961733 conn(0x7f52c406eed0 0x7f52c4071380 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f52d4009af0 tx=0x7f52d400b040 comp rx=0 tx=0).stop 2024-09-10T19:03:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:55.015+0000 7f52df582640 1 -- 172.21.15.119:0/577316877 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52d80fc1b0 msgr2=0x7f52d810c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:55.015+0000 7f52df582640 1 --2- 172.21.15.119:0/577316877 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52d80fc1b0 0x7f52d810c2e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f52c80040c0 tx=0x7f52c8009da0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:55.015+0000 7f52df582640 1 -- 172.21.15.119:0/577316877 shutdown_connections 2024-09-10T19:03:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:55.015+0000 7f52df582640 1 --2- 172.21.15.119:0/577316877 >> v2:172.21.15.119:6800/3118961733 conn(0x7f52c406eed0 0x7f52c4071380 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:55.015+0000 7f52df582640 1 --2- 172.21.15.119:0/577316877 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52d81017d0 0x7f52d810c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:55.015+0000 7f52df582640 1 --2- 172.21.15.119:0/577316877 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f52d80fc1b0 0x7f52d810c2e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:55.015+0000 7f52df582640 1 --2- 172.21.15.119:0/577316877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52d80fb800 0x7f52d810bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:55.015+0000 7f52df582640 1 -- 172.21.15.119:0/577316877 >> 172.21.15.119:0/577316877 conn(0x7f52d8071c00 msgr2=0x7f52d80fa220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:00.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:55.015+0000 7f52df582640 1 -- 172.21.15.119:0/577316877 shutdown_connections 2024-09-10T19:03:00.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:02:55.015+0000 7f52df582640 1 -- 172.21.15.119:0/577316877 wait complete. 2024-09-10T19:03:00.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.239+0000 7fa18dce0640 1 -- 172.21.15.119:0/2982993139 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa18810ad10 con 0x7fa1880fb810 2024-09-10T19:03:00.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.239+0000 7fa16ffff640 1 -- 172.21.15.119:0/2982993139 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa17805d050 con 0x7fa1880fb810 2024-09-10T19:03:00.240 INFO:teuthology.orchestra.run.smithi119.stdout:34359738421 2024-09-10T19:03:00.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.241+0000 7fa18dce0640 1 -- 172.21.15.119:0/2982993139 >> v2:172.21.15.119:6800/3118961733 conn(0x7fa17006eed0 msgr2=0x7fa170071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.241+0000 7fa18dce0640 1 --2- 172.21.15.119:0/2982993139 >> v2:172.21.15.119:6800/3118961733 conn(0x7fa17006eed0 0x7fa170071380 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa17c009f80 tx=0x7fa17c009da0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.241+0000 7fa18dce0640 1 -- 172.21.15.119:0/2982993139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1880fb810 msgr2=0x7fa188111cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.241+0000 7fa18dce0640 1 --2- 172.21.15.119:0/2982993139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1880fb810 0x7fa188111cc0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa178010e00 tx=0x7fa17800b040 comp rx=0 tx=0).stop 2024-09-10T19:03:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.241+0000 7fa18dce0640 1 -- 172.21.15.119:0/2982993139 shutdown_connections 2024-09-10T19:03:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.241+0000 7fa18dce0640 1 --2- 172.21.15.119:0/2982993139 >> v2:172.21.15.119:6800/3118961733 conn(0x7fa17006eed0 0x7fa170071380 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.241+0000 7fa18dce0640 1 --2- 172.21.15.119:0/2982993139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa188101830 0x7fa18810be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.241+0000 7fa18dce0640 1 --2- 172.21.15.119:0/2982993139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1880fc1c0 0x7fa188112200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.241+0000 7fa18dce0640 1 --2- 172.21.15.119:0/2982993139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa1880fb810 0x7fa188111cc0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.241+0000 7fa18dce0640 1 -- 172.21.15.119:0/2982993139 >> 172.21.15.119:0/2982993139 conn(0x7fa1880713f0 msgr2=0x7fa188072690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.241+0000 7fa18dce0640 1 -- 172.21.15.119:0/2982993139 shutdown_connections 2024-09-10T19:03:00.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:00.241+0000 7fa18dce0640 1 -- 172.21.15.119:0/2982993139 wait complete. 2024-09-10T19:03:00.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:00 smithi194 ceph-mon[32464]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:00.268 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:03:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:03:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:03:00.499 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:00 smithi119 ceph-mon[29577]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:00.499 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:00 smithi119 ceph-mon[40674]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:00.831 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430224 got 206158430228 for osd.7 2024-09-10T19:03:00.831 DEBUG:teuthology.parallel:result is None 2024-09-10T19:03:01.392 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2982993139' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T19:03:01.392 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2982993139' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T19:03:01.481 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2982993139' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T19:03:01.683 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:01 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[75363]: ts=2024-09-10T19:03:01.572Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00158657s 2024-09-10T19:03:01.708 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738417 got 34359738421 for osd.0 2024-09-10T19:03:01.708 DEBUG:teuthology.parallel:result is None 2024-09-10T19:03:01.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.880+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2880542351 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc1028e0 msgr2=0x7f1efc102cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:01.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.880+0000 7f1f01b7c640 1 --2- 172.21.15.119:0/2880542351 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc1028e0 0x7f1efc102cb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1eec004950 tx=0x7f1eec009da0 comp rx=0 tx=0).stop 2024-09-10T19:03:01.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.880+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2880542351 shutdown_connections 2024-09-10T19:03:01.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.880+0000 7f1f01b7c640 1 --2- 172.21.15.119:0/2880542351 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1efc0fd3e0 0x7f1efc109630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:01.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.880+0000 7f1f01b7c640 1 --2- 172.21.15.119:0/2880542351 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1efc0fca50 0x7f1efc0fcea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:01.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.880+0000 7f1f01b7c640 1 --2- 172.21.15.119:0/2880542351 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc1028e0 0x7f1efc102cb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:01.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.880+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2880542351 >> 172.21.15.119:0/2880542351 conn(0x7f1efc073660 msgr2=0x7f1efc073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:01.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.880+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2880542351 shutdown_connections 2024-09-10T19:03:01.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.880+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2880542351 wait complete. 2024-09-10T19:03:01.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1f01b7c640 1 Processor -- start 2024-09-10T19:03:01.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1f01b7c640 1 -- start start 2024-09-10T19:03:01.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1f01b7c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1efc0fca50 0x7f1efc112d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:01.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1f01b7c640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1efc0fd3e0 0x7f1efc113290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:01.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1f01b7c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc1028e0 0x7f1efc10cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:01.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1f01b7c640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc10d530 con 0x7f1efc0fca50 2024-09-10T19:03:01.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1f01b7c640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc10d6a0 con 0x7f1efc0fd3e0 2024-09-10T19:03:01.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1f01b7c640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc10d860 con 0x7f1efc1028e0 2024-09-10T19:03:01.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1efbfff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc1028e0 0x7f1efc10cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:01.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1efb7fe640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1efc0fca50 0x7f1efc112d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:01.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1efbfff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc1028e0 0x7f1efc10cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:50474/0 (socket says 172.21.15.119:50474) 2024-09-10T19:03:01.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1efbfff640 1 -- 172.21.15.119:0/2290826645 learned_addr learned my addr 172.21.15.119:0/2290826645 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:03:01.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1efaffd640 1 --2- 172.21.15.119:0/2290826645 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1efc0fd3e0 0x7f1efc113290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:01.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1efbfff640 1 -- 172.21.15.119:0/2290826645 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1efc0fd3e0 msgr2=0x7f1efc113290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:01.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1efbfff640 1 --2- 172.21.15.119:0/2290826645 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1efc0fd3e0 0x7f1efc113290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:01.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1efbfff640 1 -- 172.21.15.119:0/2290826645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1efc0fca50 msgr2=0x7f1efc112d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:01.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1efbfff640 1 --2- 172.21.15.119:0/2290826645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1efc0fca50 0x7f1efc112d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:01.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1efbfff640 1 -- 172.21.15.119:0/2290826645 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef40098b0 con 0x7f1efc1028e0 2024-09-10T19:03:01.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.881+0000 7f1efb7fe640 1 --2- 172.21.15.119:0/2290826645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1efc0fca50 0x7f1efc112d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:03:01.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.882+0000 7f1efbfff640 1 --2- 172.21.15.119:0/2290826645 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc1028e0 0x7f1efc10cee0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4001eb0 tx=0x7f1ef4004c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:03:01.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.882+0000 7f1ee3fff640 1 -- 172.21.15.119:0/2290826645 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef400fea0 con 0x7f1efc1028e0 2024-09-10T19:03:01.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.882+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2290826645 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec005040 con 0x7f1efc1028e0 2024-09-10T19:03:01.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.882+0000 7f1ee3fff640 1 -- 172.21.15.119:0/2290826645 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f1ef4007d00 con 0x7f1efc1028e0 2024-09-10T19:03:01.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.882+0000 7f1ee3fff640 1 -- 172.21.15.119:0/2290826645 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4015470 con 0x7f1efc1028e0 2024-09-10T19:03:01.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.882+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2290826645 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1efc119ba0 con 0x7f1efc1028e0 2024-09-10T19:03:01.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.883+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2290826645 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1efc0fe000 con 0x7f1efc1028e0 2024-09-10T19:03:01.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.883+0000 7f1ee3fff640 1 -- 172.21.15.119:0/2290826645 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f1ef4007890 con 0x7f1efc1028e0 2024-09-10T19:03:01.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.883+0000 7f1ee3fff640 1 --2- 172.21.15.119:0/2290826645 >> v2:172.21.15.119:6800/3118961733 conn(0x7f1ee406eed0 0x7f1ee4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:01.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.884+0000 7f1ee3fff640 1 -- 172.21.15.119:0/2290826645 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f1ef408e520 con 0x7f1efc1028e0 2024-09-10T19:03:01.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.884+0000 7f1efb7fe640 1 --2- 172.21.15.119:0/2290826645 >> v2:172.21.15.119:6800/3118961733 conn(0x7f1ee406eed0 0x7f1ee4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:01.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.884+0000 7f1efb7fe640 1 --2- 172.21.15.119:0/2290826645 >> v2:172.21.15.119:6800/3118961733 conn(0x7f1ee406eed0 0x7f1ee4071380 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1eec0040d0 tx=0x7f1eec009da0 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:03:01.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:01.888+0000 7f1ee3fff640 1 -- 172.21.15.119:0/2290826645 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f1ef405b430 con 0x7f1efc1028e0 2024-09-10T19:03:01.898 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:02.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:02.052+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2290826645 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1efc10be40 con 0x7f1efc1028e0 2024-09-10T19:03:02.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:02.052+0000 7f1ee3fff640 1 -- 172.21.15.119:0/2290826645 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f1ef405add0 con 0x7f1efc1028e0 2024-09-10T19:03:02.054 INFO:teuthology.orchestra.run.smithi119.stdout:107374182440 2024-09-10T19:03:02.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:02.055+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2290826645 >> v2:172.21.15.119:6800/3118961733 conn(0x7f1ee406eed0 msgr2=0x7f1ee4071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:02.055+0000 7f1f01b7c640 1 --2- 172.21.15.119:0/2290826645 >> v2:172.21.15.119:6800/3118961733 conn(0x7f1ee406eed0 0x7f1ee4071380 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1eec0040d0 tx=0x7f1eec009da0 comp rx=0 tx=0).stop 2024-09-10T19:03:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:02.055+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2290826645 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc1028e0 msgr2=0x7f1efc10cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:02.055+0000 7f1f01b7c640 1 --2- 172.21.15.119:0/2290826645 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc1028e0 0x7f1efc10cee0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4001eb0 tx=0x7f1ef4004c50 comp rx=0 tx=0).stop 2024-09-10T19:03:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:02.055+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2290826645 shutdown_connections 2024-09-10T19:03:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:02.055+0000 7f1f01b7c640 1 --2- 172.21.15.119:0/2290826645 >> v2:172.21.15.119:6800/3118961733 conn(0x7f1ee406eed0 0x7f1ee4071380 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:02.055+0000 7f1f01b7c640 1 --2- 172.21.15.119:0/2290826645 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc1028e0 0x7f1efc10cee0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:02.055+0000 7f1f01b7c640 1 --2- 172.21.15.119:0/2290826645 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1efc0fd3e0 0x7f1efc113290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:02.055+0000 7f1f01b7c640 1 --2- 172.21.15.119:0/2290826645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1efc0fca50 0x7f1efc112d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:02.055+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2290826645 >> 172.21.15.119:0/2290826645 conn(0x7f1efc073660 msgr2=0x7f1efc0fb620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:02.055+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2290826645 shutdown_connections 2024-09-10T19:03:02.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:02.055+0000 7f1f01b7c640 1 -- 172.21.15.119:0/2290826645 wait complete. 2024-09-10T19:03:02.170 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 podman[65785]: 2024-09-10 19:03:02.106661199 +0000 UTC m=+0.123486663 container create f222ca35a17b7d11dd64c78504b7d0f7961ec581985bb3fd34ce5372dbae6d41 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a, name=ubi8, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, architecture=x86_64, io.openshift.tags=base rhel8, com.redhat.component=ubi8-container, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, description=Ceph Grafana Container, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner , build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0) 2024-09-10T19:03:02.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:02 smithi119 ceph-mon[29577]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:02.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2290826645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T19:03:02.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:02.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:02 smithi119 ceph-mon[40674]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:02.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2290826645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T19:03:02.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:02.452 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-mon[32464]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:02.452 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2290826645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T19:03:02.452 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:02.453 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 podman[65785]: 2024-09-10 19:03:02.342455278 +0000 UTC m=+0.359280700 container init f222ca35a17b7d11dd64c78504b7d0f7961ec581985bb3fd34ce5372dbae6d41 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base rhel8, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, release=239.1651231664, vcs-type=git, com.redhat.component=ubi8-container, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5, description=Ceph Grafana Container, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0) 2024-09-10T19:03:02.453 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 podman[65785]: 2024-09-10 19:03:02.347391463 +0000 UTC m=+0.364216885 container start f222ca35a17b7d11dd64c78504b7d0f7961ec581985bb3fd34ce5372dbae6d41 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, com.redhat.component=ubi8-container, io.openshift.expose-services=, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, vcs-type=git, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, version=8.5) 2024-09-10T19:03:02.704 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 bash[65785]: f222ca35a17b7d11dd64c78504b7d0f7961ec581985bb3fd34ce5372dbae6d41 2024-09-10T19:03:02.704 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 systemd[1]: Started Ceph grafana.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:03:02.704 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=settings t=2024-09-10T19:03:02.501813594Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-10T19:03:02.704 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=settings t=2024-09-10T19:03:02.502051613Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-10T19:03:02.705 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=settings t=2024-09-10T19:03:02.502078847Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-10T19:03:02.705 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=settings t=2024-09-10T19:03:02.502092753Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-10T19:03:02.705 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=settings t=2024-09-10T19:03:02.502103787Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-10T19:03:02.705 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=settings t=2024-09-10T19:03:02.50211484Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-10T19:03:02.705 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=settings t=2024-09-10T19:03:02.502125413Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-10T19:03:02.705 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=settings t=2024-09-10T19:03:02.50213612Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-10T19:03:02.705 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=settings t=2024-09-10T19:03:02.50214694Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-10T19:03:02.705 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=settings t=2024-09-10T19:03:02.502157219Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-10T19:03:02.705 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=settings t=2024-09-10T19:03:02.502167241Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-10T19:03:02.705 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=settings t=2024-09-10T19:03:02.502177308Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-10T19:03:02.705 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=settings t=2024-09-10T19:03:02.502188826Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-10T19:03:02.705 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=settings t=2024-09-10T19:03:02.502200606Z level=info msg="App mode production" 2024-09-10T19:03:02.705 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=sqlstore t=2024-09-10T19:03:02.502264519Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-10T19:03:02.706 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=sqlstore t=2024-09-10T19:03:02.50229264Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-10T19:03:02.706 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.503170592Z level=info msg="Starting DB migrations" 2024-09-10T19:03:02.706 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.503255439Z level=info msg="Executing migration" id="create migration_log table" 2024-09-10T19:03:02.707 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.503905862Z level=info msg="Executing migration" id="create user table" 2024-09-10T19:03:02.707 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.504359638Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-10T19:03:02.707 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.507147373Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-10T19:03:02.707 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.509822607Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-10T19:03:02.707 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.512463429Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-10T19:03:02.707 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.515065828Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-10T19:03:02.707 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.517779942Z level=info msg="Executing migration" id="create user table v2" 2024-09-10T19:03:02.708 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.518309584Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-10T19:03:02.708 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.521150563Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-10T19:03:02.708 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.521906769Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-10T19:03:02.708 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.522734124Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-10T19:03:02.708 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.523520756Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-10T19:03:02.708 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.52431312Z level=info msg="Executing migration" id="Update user table charset" 2024-09-10T19:03:02.708 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.525152277Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-10T19:03:02.708 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.525557542Z level=info msg="Executing migration" id="Add missing user data" 2024-09-10T19:03:02.708 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.526422134Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-10T19:03:02.708 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.527231545Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-10T19:03:02.708 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.528081491Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-10T19:03:02.708 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.528490962Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-10T19:03:02.709 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.529593013Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-10T19:03:02.709 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.530505615Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-10T19:03:02.709 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.530932686Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-10T19:03:02.709 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.531331695Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-10T19:03:02.709 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.531707548Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-10T19:03:02.709 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.532076943Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-10T19:03:02.709 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.532365692Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-10T19:03:02.709 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.532704882Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-10T19:03:02.709 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.533047399Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-10T19:03:02.709 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.533432067Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-10T19:03:02.709 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.533768141Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-10T19:03:02.709 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.536397539Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-10T19:03:02.709 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.536808348Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-10T19:03:02.710 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.53723065Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-10T19:03:02.710 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.540016676Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-10T19:03:02.710 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.540432309Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-10T19:03:02.710 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.540849831Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-10T19:03:02.710 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.54119187Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-10T19:03:02.710 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.541532295Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-10T19:03:02.710 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.541851308Z level=info msg="Executing migration" id="create star table" 2024-09-10T19:03:02.710 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.542257014Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-10T19:03:02.710 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.542675751Z level=info msg="Executing migration" id="create org table v1" 2024-09-10T19:03:02.710 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.543061422Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-10T19:03:02.710 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.543471435Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-10T19:03:02.710 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.543852976Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-10T19:03:02.710 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.54428689Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-10T19:03:02.710 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.544712175Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-10T19:03:02.711 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.545119773Z level=info msg="Executing migration" id="Update org table charset" 2024-09-10T19:03:02.711 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.547493337Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-10T19:03:02.711 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.550117148Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-10T19:03:02.711 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.552933974Z level=info msg="Executing migration" id="create dashboard table" 2024-09-10T19:03:02.711 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.553362696Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-10T19:03:02.711 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.556314646Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-10T19:03:02.711 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.558846706Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-10T19:03:02.711 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.561182569Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-10T19:03:02.711 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.56384419Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-10T19:03:02.711 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.566177081Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-10T19:03:02.711 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.568702549Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-10T19:03:02.711 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.571162525Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-10T19:03:02.711 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.571701754Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-10T19:03:02.711 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.574406651Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-10T19:03:02.712 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.576840072Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-10T19:03:02.712 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.579379789Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-10T19:03:02.712 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.579726441Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-10T19:03:02.712 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.58242536Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-10T19:03:02.712 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.585035553Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-10T19:03:02.712 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.587444307Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-10T19:03:02.712 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.587904385Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-10T19:03:02.712 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.590660189Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-10T19:03:02.712 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.59312372Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-10T19:03:02.712 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.593800142Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-10T19:03:02.712 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.594130517Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-10T19:03:02.712 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.595039233Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-10T19:03:02.712 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.595589922Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-10T19:03:02.713 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.596078269Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-10T19:03:02.713 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.596585432Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-10T19:03:02.713 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.597159812Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-10T19:03:02.713 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.597907947Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-10T19:03:02.713 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.598761016Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-10T19:03:02.713 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.599327928Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-10T19:03:02.713 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.599639718Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-10T19:03:02.713 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.60032461Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-10T19:03:02.713 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.600741947Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-10T19:03:02.713 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.601665534Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-10T19:03:02.713 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.602406618Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-10T19:03:02.713 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.602842291Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-10T19:03:02.714 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.603627823Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-10T19:03:02.714 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.604504813Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-10T19:03:02.714 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.60520499Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-10T19:03:02.714 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.605784873Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-10T19:03:02.714 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.60662027Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-10T19:03:02.714 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.607453256Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-10T19:03:02.714 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.608195116Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-10T19:03:02.714 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.608640844Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-10T19:03:02.714 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.609409333Z level=info msg="Executing migration" id="create data_source table" 2024-09-10T19:03:02.714 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.610240525Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-10T19:03:02.714 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.610695124Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-10T19:03:02.714 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.611135467Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-10T19:03:02.714 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.611942519Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.612703268Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.613714033Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.614136485Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.614919455Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.615708624Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.616460849Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.617039972Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.617586728Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.620273548Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.620620969Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.623020883Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.625869126Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.626253086Z level=info msg="Executing migration" id="Add uid column" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.628988076Z level=info msg="Executing migration" id="Update uid value" 2024-09-10T19:03:02.715 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.629306948Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-10T19:03:02.716 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.631449974Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-10T19:03:02.716 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.631904705Z level=info msg="Executing migration" id="create api_key table" 2024-09-10T19:03:02.716 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.634138194Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-10T19:03:02.716 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.634548989Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-10T19:03:02.716 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.637148497Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-10T19:03:02.716 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.640004429Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-10T19:03:02.716 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.642630057Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-10T19:03:02.716 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.645266191Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-10T19:03:02.716 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.64821367Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-10T19:03:02.716 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.651008615Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-10T19:03:02.716 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.651567393Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-10T19:03:02.716 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.651953066Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-10T19:03:02.716 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.652382887Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-10T19:03:02.717 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.652796825Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-10T19:03:02.717 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.653135857Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-10T19:03:02.717 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.65348937Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-10T19:03:02.717 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.653751507Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-10T19:03:02.717 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.654306689Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-10T19:03:02.717 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.654842804Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-10T19:03:02.717 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.655176192Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-10T19:03:02.717 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.655720498Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-10T19:03:02.717 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.656278377Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-10T19:03:02.718 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.656670287Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-10T19:03:02.718 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.657030707Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-10T19:03:02.718 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.657425027Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-10T19:03:02.718 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.65783603Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-10T19:03:02.718 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.658274782Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-10T19:03:02.718 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.65869432Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-10T19:03:02.718 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.658971664Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-10T19:03:02.718 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.659281803Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-10T19:03:02.718 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.659883351Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-10T19:03:02.718 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.660488651Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-10T19:03:02.718 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.660818915Z level=info msg="Executing migration" id="create quota table v1" 2024-09-10T19:03:02.718 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.661226163Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.66167208Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.661945156Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.662451Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.662886049Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.663468205Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.663777978Z level=info msg="Executing migration" id="create session table" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.664224174Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.664516617Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.66480205Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.665206485Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.665590008Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.66585751Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.666156407Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.666709286Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-10T19:03:02.719 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.667535096Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.668002497Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.668335944Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.669157522Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.669489412Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.670052084Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.670684174Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.670988652Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.671431493Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.671920251Z level=info msg="Executing migration" id="create alert table v1" 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.672768513Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.673756448Z level=info msg="Executing migration" id="add index alert state" 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.674460041Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.675356745Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.676169655Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-10T19:03:02.720 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.67705824Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.678024741Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.679455811Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.680332862Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.68131748Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.68215531Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.682934508Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.683931808Z level=info msg="Executing migration" id="Add column is_default" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.686737416Z level=info msg="Executing migration" id="Add column frequency" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.689560684Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.692406648Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.695133158Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.698011495Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.700632662Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.703561919Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-10T19:03:02.721 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.706337765Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-10T19:03:02.873 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182436 got 107374182440 for osd.3 2024-09-10T19:03:02.874 DEBUG:teuthology.parallel:result is None 2024-09-10T19:03:02.955 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.709254077Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-10T19:03:02.955 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.711999406Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-10T19:03:02.955 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.714845805Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-10T19:03:02.955 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.717744427Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-10T19:03:02.955 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.720614855Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-10T19:03:02.955 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.723513916Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-10T19:03:02.955 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.726215833Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-10T19:03:02.956 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.729074096Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-10T19:03:02.956 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.731841307Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-10T19:03:02.956 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.734785315Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-10T19:03:02.956 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.737540694Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-10T19:03:02.956 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.740149517Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-10T19:03:02.956 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.742877919Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-10T19:03:02.956 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.745763719Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-10T19:03:02.956 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.748494102Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-10T19:03:02.956 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.751074663Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-10T19:03:02.956 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.75386208Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-10T19:03:02.956 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.756645052Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-10T19:03:02.956 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.759370434Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-10T19:03:02.957 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.76214298Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-10T19:03:02.957 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.76481264Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-10T19:03:02.957 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.767449173Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-10T19:03:02.958 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.770319647Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-10T19:03:02.958 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.772809157Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-10T19:03:02.958 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.775615709Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-10T19:03:02.958 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.778198869Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-10T19:03:02.958 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.781089837Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-10T19:03:02.958 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.782745839Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-10T19:03:02.958 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.783546807Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-10T19:03:02.958 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.784526587Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-10T19:03:02.958 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.78516221Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-10T19:03:02.958 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.786155388Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-10T19:03:02.958 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.786854787Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-10T19:03:02.958 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.787821965Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-10T19:03:02.958 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.788662232Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-10T19:03:02.959 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.789486136Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-10T19:03:02.959 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.789995919Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-10T19:03:02.959 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.790308333Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-10T19:03:02.959 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.791177418Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-10T19:03:02.959 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.791607408Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-10T19:03:02.959 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.792023596Z level=info msg="Executing migration" id="Move region to single row" 2024-09-10T19:03:02.959 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.792391058Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-10T19:03:02.959 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.792877185Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-10T19:03:02.959 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.793292371Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-10T19:03:02.959 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.793733584Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-10T19:03:02.959 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.794614129Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-10T19:03:02.959 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.795585912Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-10T19:03:02.959 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.796352897Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-10T19:03:02.959 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.797156547Z level=info msg="Executing migration" id="create test_data table" 2024-09-10T19:03:02.960 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.798189558Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-10T19:03:02.960 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.798895477Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-10T19:03:02.960 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.799869268Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-10T19:03:02.960 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.800600318Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-10T19:03:02.960 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.80148598Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-10T19:03:02.960 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.801978266Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-10T19:03:02.960 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.802343175Z level=info msg="Executing migration" id="create team table" 2024-09-10T19:03:02.960 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.803291976Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-10T19:03:02.960 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.804157071Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-10T19:03:02.960 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.804870803Z level=info msg="Executing migration" id="create team member table" 2024-09-10T19:03:02.960 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.805836175Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-10T19:03:02.960 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.80657457Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-10T19:03:02.960 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.807545244Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-10T19:03:02.961 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.81031887Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-10T19:03:02.961 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.813074244Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-10T19:03:02.961 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.816086636Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-10T19:03:02.961 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.818692053Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-10T19:03:02.961 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.821601885Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-10T19:03:02.961 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.824214624Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-10T19:03:02.961 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.826855067Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-10T19:03:02.961 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.829736608Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-10T19:03:02.961 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.83255729Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-10T19:03:02.961 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.835415429Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-10T19:03:02.961 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.835988656Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-10T19:03:02.961 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.838834586Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-10T19:03:02.961 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.841704876Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-10T19:03:02.962 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.84438962Z level=info msg="Executing migration" id="create tag table" 2024-09-10T19:03:02.962 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.844948779Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-10T19:03:02.962 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.847893797Z level=info msg="Executing migration" id="create login attempt table" 2024-09-10T19:03:02.962 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.850769582Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-10T19:03:02.962 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.853600325Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-10T19:03:02.962 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.856330312Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-10T19:03:02.962 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.859152572Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-10T19:03:02.962 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.861780741Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-10T19:03:02.962 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.864532644Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-10T19:03:02.962 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.867118615Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-10T19:03:02.962 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.869799794Z level=info msg="Executing migration" id="create user auth table" 2024-09-10T19:03:02.962 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.872604226Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-10T19:03:02.962 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.875211982Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-10T19:03:02.962 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.877986171Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-10T19:03:02.963 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.880699237Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-10T19:03:02.963 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.883559161Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-10T19:03:02.963 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.886150331Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-10T19:03:02.963 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.888865472Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-10T19:03:02.963 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.891620177Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-10T19:03:02.963 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.894130184Z level=info msg="Executing migration" id="create server_lock table" 2024-09-10T19:03:02.963 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.897082045Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-10T19:03:02.963 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.899759805Z level=info msg="Executing migration" id="create user auth token table" 2024-09-10T19:03:02.963 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.900574242Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-10T19:03:02.963 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.901271912Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-10T19:03:02.963 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.902092185Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-10T19:03:02.963 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.902874711Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-10T19:03:02.963 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.903723674Z level=info msg="Executing migration" id="create cache_data table" 2024-09-10T19:03:02.964 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.904729666Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-10T19:03:02.964 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.90550599Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-10T19:03:02.964 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.905925559Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-10T19:03:02.964 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.908452643Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-10T19:03:02.964 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.908861955Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-10T19:03:02.964 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.909168097Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-10T19:03:02.964 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.909584148Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-10T19:03:02.964 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.912281238Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-10T19:03:02.964 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.912888099Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-10T19:03:02.964 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.913188004Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-10T19:03:02.964 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.913583921Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-10T19:03:02.965 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.913966303Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-10T19:03:02.965 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.914431329Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-10T19:03:02.965 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.914841171Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-10T19:03:02.965 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.915643067Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-10T19:03:02.965 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.916064873Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-10T19:03:02.965 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.916360086Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-10T19:03:02.965 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.916727755Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-10T19:03:02.965 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.917150897Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-10T19:03:02.965 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.917557701Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-10T19:03:02.965 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.917816605Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-10T19:03:02.965 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.91823841Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.918630976Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.919015329Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.919416602Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.92021801Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.920619901Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.92100768Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.924385887Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.927664439Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.928114052Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.930580085Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.931457871Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.93186555Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.934469498Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.936887109Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-10T19:03:02.966 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.939225604Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-10T19:03:02.967 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.941934532Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-10T19:03:02.967 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.944420011Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-10T19:03:02.967 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.946888351Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-10T19:03:02.967 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.949283546Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-10T19:03:02.967 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.951830482Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-10T19:03:02.967 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.954362664Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-10T19:03:02.967 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.957011457Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-10T19:03:03.247 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.959595352Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-10T19:03:03.247 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.960385246Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-10T19:03:03.247 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.961239983Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-10T19:03:03.247 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.962073305Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-10T19:03:03.247 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.962867581Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-10T19:03:03.247 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.963654812Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-10T19:03:03.247 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.964434728Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-10T19:03:03.247 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.965168412Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-10T19:03:03.247 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.965493261Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-10T19:03:03.247 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.966401059Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-10T19:03:03.248 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.967268291Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-10T19:03:03.248 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.968141229Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-10T19:03:03.248 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.969005371Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-10T19:03:03.248 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.969867975Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-10T19:03:03.248 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.970673419Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-10T19:03:03.248 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.971474003Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-10T19:03:03.248 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.972379222Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-10T19:03:03.249 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.97320046Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-10T19:03:03.249 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.974109048Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-10T19:03:03.249 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.974993835Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-10T19:03:03.249 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.975854121Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-10T19:03:03.249 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.976781163Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-10T19:03:03.249 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.977436671Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-10T19:03:03.249 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.97829422Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-10T19:03:03.249 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.97906793Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-10T19:03:03.249 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.979851487Z level=info msg="Executing migration" id="create alert_image table" 2024-09-10T19:03:03.249 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.980664769Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-10T19:03:03.250 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.981565149Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-10T19:03:03.250 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.98221435Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-10T19:03:03.250 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.983063796Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-10T19:03:03.250 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.983661529Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-10T19:03:03.250 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.983732123Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-10T19:03:03.250 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.984413499Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-10T19:03:03.250 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.984794988Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-10T19:03:03.250 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.98523092Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-10T19:03:03.250 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.985344326Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-10T19:03:03.250 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.985450872Z level=info msg="alerts found to migrate" alerts=0 2024-09-10T19:03:03.250 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.986029146Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-10T19:03:03.250 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.986841299Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-10T19:03:03.251 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.987633766Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-10T19:03:03.251 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.988550718Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-10T19:03:03.251 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.991102233Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-10T19:03:03.251 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.993718601Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-10T19:03:03.251 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.99661442Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-10T19:03:03.251 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:02.999287673Z level=info msg="Executing migration" id="create data_keys table" 2024-09-10T19:03:03.251 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.00200443Z level=info msg="Executing migration" id="create secrets table" 2024-09-10T19:03:03.251 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.004692126Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-10T19:03:03.251 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.008957646Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-10T19:03:03.251 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.011734029Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-10T19:03:03.251 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.014339456Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-10T19:03:03.251 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.018627133Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-10T19:03:03.251 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.022856279Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-10T19:03:03.251 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.025488998Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-10T19:03:03.252 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.028231624Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-10T19:03:03.252 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.028573007Z level=info msg="Executing migration" id="create permission table" 2024-09-10T19:03:03.252 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.030884911Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-10T19:03:03.252 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.033673574Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-10T19:03:03.252 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.036622782Z level=info msg="Executing migration" id="create role table" 2024-09-10T19:03:03.252 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.039274499Z level=info msg="Executing migration" id="add column display_name" 2024-09-10T19:03:03.252 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.042003724Z level=info msg="Executing migration" id="add column group_name" 2024-09-10T19:03:03.252 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.044443618Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-10T19:03:03.252 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.047035147Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-10T19:03:03.252 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.049406651Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-10T19:03:03.252 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.052001174Z level=info msg="Executing migration" id="create team role table" 2024-09-10T19:03:03.252 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.054485039Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-10T19:03:03.252 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.056820563Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-10T19:03:03.253 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.059311829Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-10T19:03:03.253 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.06205085Z level=info msg="Executing migration" id="create user role table" 2024-09-10T19:03:03.253 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.064373353Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-10T19:03:03.253 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.066870578Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-10T19:03:03.253 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.069310148Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-10T19:03:03.253 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.071749507Z level=info msg="Executing migration" id="create builtin role table" 2024-09-10T19:03:03.253 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.074233658Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-10T19:03:03.253 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.076781699Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-10T19:03:03.254 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.079599595Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-10T19:03:03.254 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.082110718Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-10T19:03:03.254 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.084618837Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-10T19:03:03.254 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.085496462Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-10T19:03:03.254 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.086072205Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-10T19:03:03.254 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.086493461Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-10T19:03:03.254 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.086861898Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-10T19:03:03.254 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.087295245Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-10T19:03:03.255 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.088272716Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-10T19:03:03.255 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.088976813Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-10T19:03:03.255 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.089403471Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-10T19:03:03.255 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.089659238Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-10T19:03:03.255 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.090033391Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-10T19:03:03.255 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.090483032Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-10T19:03:03.255 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.090809306Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-10T19:03:03.255 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.091114271Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-10T19:03:03.255 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.091458945Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-10T19:03:03.255 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.091832826Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-10T19:03:03.255 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.092298305Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-10T19:03:03.255 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.093252118Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-10T19:03:03.255 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.093546514Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-10T19:03:03.255 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.093944592Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-10T19:03:03.256 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.094367245Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-10T19:03:03.256 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.094772829Z level=info msg="Executing migration" id="add correlation config column" 2024-09-10T19:03:03.256 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.095756908Z level=info msg="Executing migration" id="create entity_events table" 2024-09-10T19:03:03.256 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.096598959Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-10T19:03:03.256 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.097608537Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-10T19:03:03.256 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.097823543Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-10T19:03:03.256 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.098295541Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-10T19:03:03.256 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.098394986Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-10T19:03:03.256 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.099129049Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-10T19:03:03.256 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.099990761Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-10T19:03:03.256 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.100793218Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-10T19:03:03.256 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.101606271Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-10T19:03:03.256 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.102474074Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-10T19:03:03.257 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.103220952Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-10T19:03:03.257 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.104053506Z level=info msg="Executing migration" id="Drop public config table" 2024-09-10T19:03:03.257 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.104903963Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-10T19:03:03.257 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.105752503Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-10T19:03:03.257 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.106598374Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-10T19:03:03.257 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.107468498Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-10T19:03:03.257 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.108429875Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-10T19:03:03.257 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.111602323Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-10T19:03:03.257 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.112620947Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-10T19:03:03.257 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.113665933Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-10T19:03:03.257 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.114777378Z level=info msg="Executing migration" id="add share column" 2024-09-10T19:03:03.258 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.115801666Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-10T19:03:03.258 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.116685183Z level=info msg="Executing migration" id="create file table" 2024-09-10T19:03:03.258 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.117559138Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-10T19:03:03.258 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.118450598Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-10T19:03:03.258 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.11928341Z level=info msg="Executing migration" id="create file_meta table" 2024-09-10T19:03:03.258 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.120231279Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-10T19:03:03.258 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.120988613Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-10T19:03:03.258 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.121843764Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-10T19:03:03.258 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.122746983Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-10T19:03:03.258 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.123688558Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-10T19:03:03.258 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.12444502Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-10T19:03:03.258 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.127350863Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-10T19:03:03.258 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.130125463Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-10T19:03:03.259 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.133000963Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-10T19:03:03.259 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.135798413Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-10T19:03:03.259 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.138608281Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-10T19:03:03.259 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.141270497Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-10T19:03:03.259 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.144041277Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-10T19:03:03.259 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.146740852Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-10T19:03:03.259 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.149512176Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-10T19:03:03.259 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.159028174Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-10T19:03:03.259 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.161956452Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-10T19:03:03.259 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.164842295Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-10T19:03:03.259 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.168651204Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-10T19:03:03.259 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.171285059Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-10T19:03:03.260 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.173928296Z level=info msg="Executing migration" id="create folder table" 2024-09-10T19:03:03.260 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.176806314Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-10T19:03:03.260 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.179551984Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-10T19:03:03.260 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.182426744Z level=info msg="Executing migration" id="Update folder title length" 2024-09-10T19:03:03.260 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.185166566Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-10T19:03:03.260 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=migrator t=2024-09-10T19:03:03.187867919Z level=info msg="migrations completed" performed=485 skipped=0 duration=684.630642ms 2024-09-10T19:03:03.260 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=sqlstore t=2024-09-10T19:03:03.188436761Z level=info msg="Created default organization" 2024-09-10T19:03:03.260 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=plugin.loader t=2024-09-10T19:03:03.245598573Z level=info msg="Plugin registered" pluginID=input 2024-09-10T19:03:03.518 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=plugin.loader t=2024-09-10T19:03:03.276485694Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-10T19:03:03.518 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=plugin.loader t=2024-09-10T19:03:03.276516932Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-10T19:03:03.518 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=secrets t=2024-09-10T19:03:03.276549498Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-10T19:03:03.519 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=query_data t=2024-09-10T19:03:03.27814498Z level=info msg="Query Service initialization" 2024-09-10T19:03:03.519 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=live.push_http t=2024-09-10T19:03:03.280354533Z level=info msg="Live Push Gateway initialization" 2024-09-10T19:03:03.631 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:03 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:03.632 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:03 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:03.632 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:03 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:03.632 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:03 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:03.632 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:03 smithi119 ceph-mon[29577]: Deploying daemon node-exporter.a on smithi119 2024-09-10T19:03:03.632 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:03 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:03.632 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:03 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:03.632 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:03 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:03.632 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:03 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:03.632 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:03 smithi119 ceph-mon[40674]: Deploying daemon node-exporter.a on smithi119 2024-09-10T19:03:03.993 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:03.994 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:03.994 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:03.994 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:03.994 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-mon[32464]: Deploying daemon node-exporter.a on smithi119 2024-09-10T19:03:04.268 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=infra.usagestats.collector t=2024-09-10T19:03:03.993159717Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-10T19:03:04.269 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=provisioning.datasources t=2024-09-10T19:03:03.993521324Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-10T19:03:04.269 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=provisioning.datasources t=2024-09-10T19:03:04.026122613Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-10T19:03:04.269 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=provisioning.datasources t=2024-09-10T19:03:04.05773018Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-10T19:03:04.269 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=provisioning.alerting t=2024-09-10T19:03:04.079248419Z level=info msg="starting to provision alerting" 2024-09-10T19:03:04.269 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=provisioning.alerting t=2024-09-10T19:03:04.079267157Z level=info msg="finished to provision alerting" 2024-09-10T19:03:04.269 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=grafanaStorageLogger t=2024-09-10T19:03:04.079414868Z level=info msg="storage starting" 2024-09-10T19:03:04.269 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=ngalert.state.manager t=2024-09-10T19:03:04.07951469Z level=info msg="Warming state cache for startup" 2024-09-10T19:03:04.269 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=http.server t=2024-09-10T19:03:04.08118702Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-10T19:03:04.269 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=sqlstore.transactions t=2024-09-10T19:03:04.090352081Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T19:03:04.269 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=ngalert.state.manager t=2024-09-10T19:03:04.09038322Z level=info msg="State cache has been initialized" states=0 duration=10.867102ms 2024-09-10T19:03:04.269 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=ticker t=2024-09-10T19:03:04.09043874Z level=info msg=starting first_tick=2024-09-10T19:03:10Z 2024-09-10T19:03:04.269 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=ngalert.multiorg.alertmanager t=2024-09-10T19:03:04.090455375Z level=info msg="starting MultiOrg Alertmanager" 2024-09-10T19:03:04.269 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=sqlstore.transactions t=2024-09-10T19:03:04.140664441Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T19:03:04.269 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=sqlstore.transactions t=2024-09-10T19:03:04.150970525Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-09-10T19:03:04.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 -- 172.21.15.119:0/2115077877 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6641028c0 msgr2=0x7ff664102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 --2- 172.21.15.119:0/2115077877 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6641028c0 0x7ff664102c90 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff660004980 tx=0x7ff660009da0 comp rx=0 tx=0).stop 2024-09-10T19:03:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 -- 172.21.15.119:0/2115077877 shutdown_connections 2024-09-10T19:03:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 --2- 172.21.15.119:0/2115077877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6640fd250 0x7ff664109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 --2- 172.21.15.119:0/2115077877 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6640fc8a0 0x7ff6640fcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 --2- 172.21.15.119:0/2115077877 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6641028c0 0x7ff664102c90 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 -- 172.21.15.119:0/2115077877 >> 172.21.15.119:0/2115077877 conn(0x7ff6640f8400 msgr2=0x7ff6640fa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 -- 172.21.15.119:0/2115077877 shutdown_connections 2024-09-10T19:03:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 -- 172.21.15.119:0/2115077877 wait complete. 2024-09-10T19:03:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 Processor -- start 2024-09-10T19:03:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 -- start start 2024-09-10T19:03:04.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6640fc8a0 0x7ff664112d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6640fd250 0x7ff6641132c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6641028c0 0x7ff66410cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66410d560 con 0x7ff6641028c0 2024-09-10T19:03:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff66410d6d0 con 0x7ff6640fc8a0 2024-09-10T19:03:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.282+0000 7ff66c2c2640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff66410d890 con 0x7ff6640fd250 2024-09-10T19:03:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff66985d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6640fd250 0x7ff6641132c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff66985d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6640fd250 0x7ff6641132c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:60830/0 (socket says 172.21.15.119:60830) 2024-09-10T19:03:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff66985d640 1 -- 172.21.15.119:0/3068613699 learned_addr learned my addr 172.21.15.119:0/3068613699 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:03:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff66a85f640 1 --2- 172.21.15.119:0/3068613699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6641028c0 0x7ff66410cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff66a05e640 1 --2- 172.21.15.119:0/3068613699 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6640fc8a0 0x7ff664112d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff66985d640 1 -- 172.21.15.119:0/3068613699 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6640fc8a0 msgr2=0x7ff664112d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff66985d640 1 --2- 172.21.15.119:0/3068613699 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6640fc8a0 0x7ff664112d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff66985d640 1 -- 172.21.15.119:0/3068613699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6641028c0 msgr2=0x7ff66410cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:04.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff66985d640 1 --2- 172.21.15.119:0/3068613699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6641028c0 0x7ff66410cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff66985d640 1 -- 172.21.15.119:0/3068613699 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff654005040 con 0x7ff6640fd250 2024-09-10T19:03:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff66a85f640 1 --2- 172.21.15.119:0/3068613699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6641028c0 0x7ff66410cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:03:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff66985d640 1 --2- 172.21.15.119:0/3068613699 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6640fd250 0x7ff6641132c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff654009780 tx=0x7ff654009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:03:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff65affd640 1 -- 172.21.15.119:0/3068613699 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6540099c0 con 0x7ff6640fd250 2024-09-10T19:03:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff66c2c2640 1 -- 172.21.15.119:0/3068613699 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff660005040 con 0x7ff6640fd250 2024-09-10T19:03:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff65affd640 1 -- 172.21.15.119:0/3068613699 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7ff65400f040 con 0x7ff6640fd250 2024-09-10T19:03:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff65affd640 1 -- 172.21.15.119:0/3068613699 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff654013ca0 con 0x7ff6640fd250 2024-09-10T19:03:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.283+0000 7ff66c2c2640 1 -- 172.21.15.119:0/3068613699 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff66406f940 con 0x7ff6640fd250 2024-09-10T19:03:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.284+0000 7ff66c2c2640 1 -- 172.21.15.119:0/3068613699 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6640fdfe0 con 0x7ff6640fd250 2024-09-10T19:03:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.285+0000 7ff65affd640 1 -- 172.21.15.119:0/3068613699 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7ff65400ebb0 con 0x7ff6640fd250 2024-09-10T19:03:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.285+0000 7ff65affd640 1 --2- 172.21.15.119:0/3068613699 >> v2:172.21.15.119:6800/3118961733 conn(0x7ff65006eed0 0x7ff650071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.285+0000 7ff65affd640 1 -- 172.21.15.119:0/3068613699 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7ff65408e3d0 con 0x7ff6640fd250 2024-09-10T19:03:04.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.285+0000 7ff66a05e640 1 --2- 172.21.15.119:0/3068613699 >> v2:172.21.15.119:6800/3118961733 conn(0x7ff65006eed0 0x7ff650071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:04.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.285+0000 7ff66a05e640 1 --2- 172.21.15.119:0/3068613699 >> v2:172.21.15.119:6800/3118961733 conn(0x7ff65006eed0 0x7ff650071380 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff660009f80 tx=0x7ff660009da0 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:03:04.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.289+0000 7ff65affd640 1 -- 172.21.15.119:0/3068613699 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7ff65405b2e0 con 0x7ff6640fd250 2024-09-10T19:03:04.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.469+0000 7ff66c2c2640 1 -- 172.21.15.119:0/3068613699 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff66410e420 con 0x7ff6640fd250 2024-09-10T19:03:04.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.470+0000 7ff65affd640 1 -- 172.21.15.119:0/3068613699 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff654013e20 con 0x7ff6640fd250 2024-09-10T19:03:04.471 INFO:teuthology.orchestra.run.smithi119.stdout:128849018916 2024-09-10T19:03:04.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.472+0000 7ff66c2c2640 1 -- 172.21.15.119:0/3068613699 >> v2:172.21.15.119:6800/3118961733 conn(0x7ff65006eed0 msgr2=0x7ff650071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:04.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.472+0000 7ff66c2c2640 1 --2- 172.21.15.119:0/3068613699 >> v2:172.21.15.119:6800/3118961733 conn(0x7ff65006eed0 0x7ff650071380 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff660009f80 tx=0x7ff660009da0 comp rx=0 tx=0).stop 2024-09-10T19:03:04.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.472+0000 7ff66c2c2640 1 -- 172.21.15.119:0/3068613699 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6640fd250 msgr2=0x7ff6641132c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:04.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.472+0000 7ff66c2c2640 1 --2- 172.21.15.119:0/3068613699 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6640fd250 0x7ff6641132c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff654009780 tx=0x7ff654009da0 comp rx=0 tx=0).stop 2024-09-10T19:03:04.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.472+0000 7ff66c2c2640 1 -- 172.21.15.119:0/3068613699 shutdown_connections 2024-09-10T19:03:04.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.472+0000 7ff66c2c2640 1 --2- 172.21.15.119:0/3068613699 >> v2:172.21.15.119:6800/3118961733 conn(0x7ff65006eed0 0x7ff650071380 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:04.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.472+0000 7ff66c2c2640 1 --2- 172.21.15.119:0/3068613699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6641028c0 0x7ff66410cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:04.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.472+0000 7ff66c2c2640 1 --2- 172.21.15.119:0/3068613699 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff6640fd250 0x7ff6641132c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:04.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.472+0000 7ff66c2c2640 1 --2- 172.21.15.119:0/3068613699 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6640fc8a0 0x7ff664112d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:04.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.472+0000 7ff66c2c2640 1 -- 172.21.15.119:0/3068613699 >> 172.21.15.119:0/3068613699 conn(0x7ff6640f8400 msgr2=0x7ff6640f9c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:04.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.472+0000 7ff66c2c2640 1 -- 172.21.15.119:0/3068613699 shutdown_connections 2024-09-10T19:03:04.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:04.472+0000 7ff66c2c2640 1 -- 172.21.15.119:0/3068613699 wait complete. 2024-09-10T19:03:04.766 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:04 smithi119 ceph-mon[29577]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:04.766 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3068613699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T19:03:04.766 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:04 smithi119 ceph-mon[40674]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:04.766 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3068613699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T19:03:05.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-mon[32464]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:05.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3068613699' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T19:03:05.067 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:04 smithi119 bash[76434]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-10T19:03:05.115 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018912 got 128849018916 for osd.4 2024-09-10T19:03:05.116 DEBUG:teuthology.parallel:result is None 2024-09-10T19:03:05.116 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-10T19:03:05.116 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:03:05.376 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:06 smithi119 ceph-mon[29577]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:06 smithi119 ceph-mon[40674]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:07.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:06 smithi194 ceph-mon[32464]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:07.139 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:06 smithi119 bash[76434]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-10T19:03:07.436 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:07 smithi119 bash[76434]: Writing manifest to image destination 2024-09-10T19:03:07.710 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:07 smithi119 podman[76434]: 2024-09-10 19:03:07.47745662 +0000 UTC m=+3.048262227 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-10T19:03:07.711 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:07 smithi119 podman[76434]: 2024-09-10 19:03:07.592623453 +0000 UTC m=+3.163429070 container create e8af2e4da1f4568a6277b6e396f0cf51f34155200d269b8cb57e62f93d0ea959 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:08.067 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 podman[76434]: 2024-09-10 19:03:08.014656478 +0000 UTC m=+3.585462090 container init e8af2e4da1f4568a6277b6e396f0cf51f34155200d269b8cb57e62f93d0ea959 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:08.067 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 podman[76434]: 2024-09-10 19:03:08.020788706 +0000 UTC m=+3.591594314 container start e8af2e4da1f4568a6277b6e396f0cf51f34155200d269b8cb57e62f93d0ea959 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:08.067 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.024Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-10T19:03:08.067 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.025Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-10T19:03:08.067 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.025Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-10T19:03:08.068 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.025Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-10T19:03:08.068 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.025Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-10T19:03:08.068 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.025Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-10T19:03:08.068 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-10T19:03:08.068 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=arp 2024-09-10T19:03:08.068 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-10T19:03:08.068 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-10T19:03:08.068 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-10T19:03:08.068 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-10T19:03:08.068 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-10T19:03:08.068 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-10T19:03:08.068 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-10T19:03:08.068 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-10T19:03:08.069 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=edac 2024-09-10T19:03:08.069 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-10T19:03:08.069 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-10T19:03:08.069 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-10T19:03:08.069 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-10T19:03:08.069 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-10T19:03:08.069 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-10T19:03:08.069 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-10T19:03:08.069 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-10T19:03:08.069 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-10T19:03:08.069 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-10T19:03:08.069 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-10T19:03:08.069 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-10T19:03:08.069 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-10T19:03:08.070 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-10T19:03:08.070 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-10T19:03:08.070 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-10T19:03:08.070 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=os 2024-09-10T19:03:08.070 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-10T19:03:08.070 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-10T19:03:08.070 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-10T19:03:08.070 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-10T19:03:08.070 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-10T19:03:08.071 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-10T19:03:08.071 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-10T19:03:08.071 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=stat 2024-09-10T19:03:08.071 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-10T19:03:08.071 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-10T19:03:08.071 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-10T19:03:08.071 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=time 2024-09-10T19:03:08.071 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-10T19:03:08.071 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=uname 2024-09-10T19:03:08.071 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-10T19:03:08.071 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-10T19:03:08.071 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.026Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-10T19:03:08.071 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.027Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-10T19:03:08.072 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a[76667]: ts=2024-09-10T19:03:08.027Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-10T19:03:08.348 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-mon[29577]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:08.348 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:08.348 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-mon[40674]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:08.349 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:08 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:08.349 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 bash[76434]: e8af2e4da1f4568a6277b6e396f0cf51f34155200d269b8cb57e62f93d0ea959 2024-09-10T19:03:08.349 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:03:08 smithi119 systemd[1]: Started Ceph node-exporter.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:03:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.346+0000 7fafa761b640 1 -- 172.21.15.119:0/453435886 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fafa00fad40 msgr2=0x7fafa00fb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.346+0000 7fafa761b640 1 --2- 172.21.15.119:0/453435886 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fafa00fad40 0x7fafa00fb1b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7faf9c004950 tx=0x7faf9c009da0 comp rx=0 tx=0).stop 2024-09-10T19:03:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.346+0000 7fafa761b640 1 -- 172.21.15.119:0/453435886 shutdown_connections 2024-09-10T19:03:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.346+0000 7fafa761b640 1 --2- 172.21.15.119:0/453435886 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafa00fb6f0 0x7fafa0109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.346+0000 7fafa761b640 1 --2- 172.21.15.119:0/453435886 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fafa00fad40 0x7fafa00fb1b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.346+0000 7fafa761b640 1 --2- 172.21.15.119:0/453435886 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fafa0102890 0x7fafa0102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.346+0000 7fafa761b640 1 -- 172.21.15.119:0/453435886 >> 172.21.15.119:0/453435886 conn(0x7fafa0074d90 msgr2=0x7fafa00f90d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.346+0000 7fafa761b640 1 -- 172.21.15.119:0/453435886 shutdown_connections 2024-09-10T19:03:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.346+0000 7fafa761b640 1 -- 172.21.15.119:0/453435886 wait complete. 2024-09-10T19:03:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.347+0000 7fafa761b640 1 Processor -- start 2024-09-10T19:03:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.347+0000 7fafa761b640 1 -- start start 2024-09-10T19:03:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.347+0000 7fafa761b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fafa00fad40 0x7fafa0114fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.347+0000 7fafa761b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafa00fb6f0 0x7fafa0115500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.347+0000 7fafa761b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fafa0102890 0x7fafa010f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.347+0000 7fafa761b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa010f7a0 con 0x7fafa00fb6f0 2024-09-10T19:03:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.347+0000 7fafa761b640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa010f910 con 0x7fafa00fad40 2024-09-10T19:03:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.347+0000 7fafa761b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa010fad0 con 0x7fafa0102890 2024-09-10T19:03:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.347+0000 7fafa4bb6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafa00fb6f0 0x7fafa0115500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.347+0000 7fafa5bb8640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fafa0102890 0x7fafa010f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.347+0000 7fafa4bb6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafa00fb6f0 0x7fafa0115500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40550/0 (socket says 172.21.15.119:40550) 2024-09-10T19:03:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.347+0000 7fafa4bb6640 1 -- 172.21.15.119:0/4252479407 learned_addr learned my addr 172.21.15.119:0/4252479407 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:03:08.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.348+0000 7fafa53b7640 1 --2- 172.21.15.119:0/4252479407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fafa00fad40 0x7fafa0114fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:08.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.348+0000 7fafa5bb8640 1 -- 172.21.15.119:0/4252479407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fafa00fad40 msgr2=0x7fafa0114fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:08.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.348+0000 7fafa5bb8640 1 --2- 172.21.15.119:0/4252479407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fafa00fad40 0x7fafa0114fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:08.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.348+0000 7fafa5bb8640 1 -- 172.21.15.119:0/4252479407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafa00fb6f0 msgr2=0x7fafa0115500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:08.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.348+0000 7fafa5bb8640 1 --2- 172.21.15.119:0/4252479407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafa00fb6f0 0x7fafa0115500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:08.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.348+0000 7fafa5bb8640 1 -- 172.21.15.119:0/4252479407 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf980098b0 con 0x7fafa0102890 2024-09-10T19:03:08.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.348+0000 7fafa4bb6640 1 --2- 172.21.15.119:0/4252479407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafa00fb6f0 0x7fafa0115500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:03:08.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.348+0000 7fafa5bb8640 1 --2- 172.21.15.119:0/4252479407 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fafa0102890 0x7fafa010f150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faf98001da0 tx=0x7faf98004c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:03:08.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.348+0000 7faf95ffb640 1 -- 172.21.15.119:0/4252479407 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9800fdd0 con 0x7fafa0102890 2024-09-10T19:03:08.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.348+0000 7faf95ffb640 1 -- 172.21.15.119:0/4252479407 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7faf98007d00 con 0x7fafa0102890 2024-09-10T19:03:08.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.348+0000 7fafa761b640 1 -- 172.21.15.119:0/4252479407 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf9c005040 con 0x7fafa0102890 2024-09-10T19:03:08.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.348+0000 7faf95ffb640 1 -- 172.21.15.119:0/4252479407 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf98009730 con 0x7fafa0102890 2024-09-10T19:03:08.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.348+0000 7fafa761b640 1 -- 172.21.15.119:0/4252479407 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa011be10 con 0x7fafa0102890 2024-09-10T19:03:08.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.349+0000 7fafa761b640 1 -- 172.21.15.119:0/4252479407 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa00fe060 con 0x7fafa0102890 2024-09-10T19:03:08.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.350+0000 7faf95ffb640 1 -- 172.21.15.119:0/4252479407 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7faf98007890 con 0x7fafa0102890 2024-09-10T19:03:08.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.350+0000 7faf95ffb640 1 --2- 172.21.15.119:0/4252479407 >> v2:172.21.15.119:6800/3118961733 conn(0x7faf8c06eed0 0x7faf8c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:08.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.350+0000 7faf95ffb640 1 -- 172.21.15.119:0/4252479407 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7faf9808e490 con 0x7fafa0102890 2024-09-10T19:03:08.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.350+0000 7fafa53b7640 1 --2- 172.21.15.119:0/4252479407 >> v2:172.21.15.119:6800/3118961733 conn(0x7faf8c06eed0 0x7faf8c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:08.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.351+0000 7fafa53b7640 1 --2- 172.21.15.119:0/4252479407 >> v2:172.21.15.119:6800/3118961733 conn(0x7faf8c06eed0 0x7faf8c071380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faf90004eb0 tx=0x7faf9000b040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:03:08.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.354+0000 7faf95ffb640 1 -- 172.21.15.119:0/4252479407 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7faf9805b3a0 con 0x7fafa0102890 2024-09-10T19:03:08.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.507+0000 7fafa761b640 1 -- 172.21.15.119:0/4252479407 --> v2:172.21.15.119:6800/3118961733 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fafa010be50 con 0x7faf8c06eed0 2024-09-10T19:03:08.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:03:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:03:08.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:08 smithi194 ceph-mon[32464]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:08.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:08 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:08.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.517+0000 7faf95ffb640 1 -- 172.21.15.119:0/4252479407 <== mgr.24452 v2:172.21.15.119:6800/3118961733 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+383643 (secure 0 0 0) 0x7fafa010be50 con 0x7faf8c06eed0 2024-09-10T19:03:08.522 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T19:03:08.524 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-10T19:03:08.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.520+0000 7fafa761b640 1 -- 172.21.15.119:0/4252479407 >> v2:172.21.15.119:6800/3118961733 conn(0x7faf8c06eed0 msgr2=0x7faf8c071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:08.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.520+0000 7fafa761b640 1 --2- 172.21.15.119:0/4252479407 >> v2:172.21.15.119:6800/3118961733 conn(0x7faf8c06eed0 0x7faf8c071380 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faf90004eb0 tx=0x7faf9000b040 comp rx=0 tx=0).stop 2024-09-10T19:03:08.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.520+0000 7fafa761b640 1 -- 172.21.15.119:0/4252479407 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fafa0102890 msgr2=0x7fafa010f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:08.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.520+0000 7fafa761b640 1 --2- 172.21.15.119:0/4252479407 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fafa0102890 0x7fafa010f150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7faf98001da0 tx=0x7faf98004c50 comp rx=0 tx=0).stop 2024-09-10T19:03:08.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.520+0000 7fafa761b640 1 -- 172.21.15.119:0/4252479407 shutdown_connections 2024-09-10T19:03:08.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.520+0000 7fafa761b640 1 --2- 172.21.15.119:0/4252479407 >> v2:172.21.15.119:6800/3118961733 conn(0x7faf8c06eed0 0x7faf8c071380 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:08.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.520+0000 7fafa761b640 1 --2- 172.21.15.119:0/4252479407 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fafa0102890 0x7fafa010f150 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:08.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.521+0000 7fafa761b640 1 --2- 172.21.15.119:0/4252479407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafa00fb6f0 0x7fafa0115500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:08.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.521+0000 7fafa761b640 1 --2- 172.21.15.119:0/4252479407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fafa00fad40 0x7fafa0114fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:08.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.521+0000 7fafa761b640 1 -- 172.21.15.119:0/4252479407 >> 172.21.15.119:0/4252479407 conn(0x7fafa0074d90 msgr2=0x7fafa0071bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:08.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.521+0000 7fafa761b640 1 -- 172.21.15.119:0/4252479407 shutdown_connections 2024-09-10T19:03:08.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:08.521+0000 7fafa761b640 1 -- 172.21.15.119:0/4252479407 wait complete. 2024-09-10T19:03:09.213 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:09 smithi194 bash[66065]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-10T19:03:09.330 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-10T19:03:07.037484+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":929,"num_read_kb":786,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":548,"ondisk_log_size":548,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385388,"kb_used_data":6552,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575828,"statfs":{"total":767960285184,"available":765517647872,"internally_reserved":0,"allocated":6709248,"data_stored":3631995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002761"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976690+0000","last_change":"2024-09-10T19:01:46.615905+0000","last_active":"2024-09-10T19:01:51.976690+0000","last_peered":"2024-09-10T19:01:51.976690+0000","last_clean":"2024-09-10T19:01:51.976690+0000","last_became_active":"2024-09-10T19:01:46.615569+0000","last_became_peered":"2024-09-10T19:01:46.615569+0000","last_unstale":"2024-09-10T19:01:51.976690+0000","last_undegraded":"2024-09-10T19:01:51.976690+0000","last_fullsized":"2024-09-10T19:01:51.976690+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:40:12.111111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336575+0000","last_change":"2024-09-10T19:01:40.316539+0000","last_active":"2024-09-10T19:01:51.336575+0000","last_peered":"2024-09-10T19:01:51.336575+0000","last_clean":"2024-09-10T19:01:51.336575+0000","last_became_active":"2024-09-10T19:01:40.316258+0000","last_became_peered":"2024-09-10T19:01:40.316258+0000","last_unstale":"2024-09-10T19:01:51.336575+0000","last_undegraded":"2024-09-10T19:01:51.336575+0000","last_fullsized":"2024-09-10T19:01:51.336575+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:11:06.642347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"59'15","reported_seq":41,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976906+0000","last_change":"2024-09-10T19:01:42.325766+0000","last_active":"2024-09-10T19:01:51.976906+0000","last_peered":"2024-09-10T19:01:51.976906+0000","last_clean":"2024-09-10T19:01:51.976906+0000","last_became_active":"2024-09-10T19:01:42.325498+0000","last_became_peered":"2024-09-10T19:01:42.325498+0000","last_unstale":"2024-09-10T19:01:51.976906+0000","last_undegraded":"2024-09-10T19:01:51.976906+0000","last_fullsized":"2024-09-10T19:01:51.976906+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:31:45.451249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330047+0000","last_change":"2024-09-10T19:01:44.345969+0000","last_active":"2024-09-10T19:01:51.330047+0000","last_peered":"2024-09-10T19:01:51.330047+0000","last_clean":"2024-09-10T19:01:51.330047+0000","last_became_active":"2024-09-10T19:01:44.345702+0000","last_became_peered":"2024-09-10T19:01:44.345702+0000","last_unstale":"2024-09-10T19:01:51.330047+0000","last_undegraded":"2024-09-10T19:01:51.330047+0000","last_fullsized":"2024-09-10T19:01:51.330047+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:21.166785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"52'1","reported_seq":25,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977457+0000","last_change":"2024-09-10T19:01:40.315403+0000","last_active":"2024-09-10T19:01:51.977457+0000","last_peered":"2024-09-10T19:01:51.977457+0000","last_clean":"2024-09-10T19:01:51.977457+0000","last_became_active":"2024-09-10T19:01:40.314931+0000","last_became_peered":"2024-09-10T19:01:40.314931+0000","last_unstale":"2024-09-10T19:01:51.977457+0000","last_undegraded":"2024-09-10T19:01:51.977457+0000","last_fullsized":"2024-09-10T19:01:51.977457+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:11.025210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"59'9","reported_seq":32,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330086+0000","last_change":"2024-09-10T19:01:42.324000+0000","last_active":"2024-09-10T19:01:51.330086+0000","last_peered":"2024-09-10T19:01:51.330086+0000","last_clean":"2024-09-10T19:01:51.330086+0000","last_became_active":"2024-09-10T19:01:42.323673+0000","last_became_peered":"2024-09-10T19:01:42.323673+0000","last_unstale":"2024-09-10T19:01:51.330086+0000","last_undegraded":"2024-09-10T19:01:51.330086+0000","last_fullsized":"2024-09-10T19:01:51.330086+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:53:38.000729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.613366+0000","last_change":"2024-09-10T19:01:44.347117+0000","last_active":"2024-09-10T19:01:52.613366+0000","last_peered":"2024-09-10T19:01:52.613366+0000","last_clean":"2024-09-10T19:01:52.613366+0000","last_became_active":"2024-09-10T19:01:44.346786+0000","last_became_peered":"2024-09-10T19:01:44.346786+0000","last_unstale":"2024-09-10T19:01:52.613366+0000","last_undegraded":"2024-09-10T19:01:52.613366+0000","last_fullsized":"2024-09-10T19:01:52.613366+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:47:10.722092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330149+0000","last_change":"2024-09-10T19:01:46.344623+0000","last_active":"2024-09-10T19:01:51.330149+0000","last_peered":"2024-09-10T19:01:51.330149+0000","last_clean":"2024-09-10T19:01:51.330149+0000","last_became_active":"2024-09-10T19:01:46.344322+0000","last_became_peered":"2024-09-10T19:01:46.344322+0000","last_unstale":"2024-09-10T19:01:51.330149+0000","last_undegraded":"2024-09-10T19:01:51.330149+0000","last_fullsized":"2024-09-10T19:01:51.330149+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:37.837564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474676+0000","last_change":"2024-09-10T19:01:40.317519+0000","last_active":"2024-09-10T19:01:52.474676+0000","last_peered":"2024-09-10T19:01:52.474676+0000","last_clean":"2024-09-10T19:01:52.474676+0000","last_became_active":"2024-09-10T19:01:40.317117+0000","last_became_peered":"2024-09-10T19:01:40.317117+0000","last_unstale":"2024-09-10T19:01:52.474676+0000","last_undegraded":"2024-09-10T19:01:52.474676+0000","last_fullsized":"2024-09-10T19:01:52.474676+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:00:25.740792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"59'5","reported_seq":26,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330232+0000","last_change":"2024-09-10T19:01:42.325747+0000","last_active":"2024-09-10T19:01:51.330232+0000","last_peered":"2024-09-10T19:01:51.330232+0000","last_clean":"2024-09-10T19:01:51.330232+0000","last_became_active":"2024-09-10T19:01:42.325386+0000","last_became_peered":"2024-09-10T19:01:42.325386+0000","last_unstale":"2024-09-10T19:01:51.330232+0000","last_undegraded":"2024-09-10T19:01:51.330232+0000","last_fullsized":"2024-09-10T19:01:51.330232+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:53.254045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.473269+0000","last_change":"2024-09-10T19:01:44.340522+0000","last_active":"2024-09-10T19:01:52.473269+0000","last_peered":"2024-09-10T19:01:52.473269+0000","last_clean":"2024-09-10T19:01:52.473269+0000","last_became_active":"2024-09-10T19:01:44.340252+0000","last_became_peered":"2024-09-10T19:01:44.340252+0000","last_unstale":"2024-09-10T19:01:52.473269+0000","last_undegraded":"2024-09-10T19:01:52.473269+0000","last_fullsized":"2024-09-10T19:01:52.473269+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:31:27.553639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346753+0000","last_change":"2024-09-10T19:01:46.346256+0000","last_active":"2024-09-10T19:01:52.346753+0000","last_peered":"2024-09-10T19:01:52.346753+0000","last_clean":"2024-09-10T19:01:52.346753+0000","last_became_active":"2024-09-10T19:01:46.346005+0000","last_became_peered":"2024-09-10T19:01:46.346005+0000","last_unstale":"2024-09-10T19:01:52.346753+0000","last_undegraded":"2024-09-10T19:01:52.346753+0000","last_fullsized":"2024-09-10T19:01:52.346753+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:20.283012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474391+0000","last_change":"2024-09-10T19:01:40.319607+0000","last_active":"2024-09-10T19:01:52.474391+0000","last_peered":"2024-09-10T19:01:52.474391+0000","last_clean":"2024-09-10T19:01:52.474391+0000","last_became_active":"2024-09-10T19:01:40.319363+0000","last_became_peered":"2024-09-10T19:01:40.319363+0000","last_unstale":"2024-09-10T19:01:52.474391+0000","last_undegraded":"2024-09-10T19:01:52.474391+0000","last_fullsized":"2024-09-10T19:01:52.474391+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:33.978961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"59'9","reported_seq":32,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330301+0000","last_change":"2024-09-10T19:01:42.321058+0000","last_active":"2024-09-10T19:01:51.330301+0000","last_peered":"2024-09-10T19:01:51.330301+0000","last_clean":"2024-09-10T19:01:51.330301+0000","last_became_active":"2024-09-10T19:01:42.320753+0000","last_became_peered":"2024-09-10T19:01:42.320753+0000","last_unstale":"2024-09-10T19:01:51.330301+0000","last_undegraded":"2024-09-10T19:01:51.330301+0000","last_fullsized":"2024-09-10T19:01:51.330301+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:42:38.537340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346436+0000","last_change":"2024-09-10T19:01:44.343628+0000","last_active":"2024-09-10T19:01:52.346436+0000","last_peered":"2024-09-10T19:01:52.346436+0000","last_clean":"2024-09-10T19:01:52.346436+0000","last_became_active":"2024-09-10T19:01:44.343359+0000","last_became_peered":"2024-09-10T19:01:44.343359+0000","last_unstale":"2024-09-10T19:01:52.346436+0000","last_undegraded":"2024-09-10T19:01:52.346436+0000","last_fullsized":"2024-09-10T19:01:52.346436+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:23:11.270396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335744+0000","last_change":"2024-09-10T19:01:46.347691+0000","last_active":"2024-09-10T19:01:51.335744+0000","last_peered":"2024-09-10T19:01:51.335744+0000","last_clean":"2024-09-10T19:01:51.335744+0000","last_became_active":"2024-09-10T19:01:46.347366+0000","last_became_peered":"2024-09-10T19:01:46.347366+0000","last_unstale":"2024-09-10T19:01:51.335744+0000","last_undegraded":"2024-09-10T19:01:51.335744+0000","last_fullsized":"2024-09-10T19:01:51.335744+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:01:01.695933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"59'17","reported_seq":44,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330743+0000","last_change":"2024-09-10T19:01:42.325217+0000","last_active":"2024-09-10T19:01:51.330743+0000","last_peered":"2024-09-10T19:01:51.330743+0000","last_clean":"2024-09-10T19:01:51.330743+0000","last_became_active":"2024-09-10T19:01:42.324863+0000","last_became_peered":"2024-09-10T19:01:42.324863+0000","last_unstale":"2024-09-10T19:01:51.330743+0000","last_undegraded":"2024-09-10T19:01:51.330743+0000","last_fullsized":"2024-09-10T19:01:51.330743+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:58:09.129606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474339+0000","last_change":"2024-09-10T19:01:40.320087+0000","last_active":"2024-09-10T19:01:52.474339+0000","last_peered":"2024-09-10T19:01:52.474339+0000","last_clean":"2024-09-10T19:01:52.474339+0000","last_became_active":"2024-09-10T19:01:40.319542+0000","last_became_peered":"2024-09-10T19:01:40.319542+0000","last_unstale":"2024-09-10T19:01:52.474339+0000","last_undegraded":"2024-09-10T19:01:52.474339+0000","last_fullsized":"2024-09-10T19:01:52.474339+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:59.316079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.613469+0000","last_change":"2024-09-10T19:01:44.345228+0000","last_active":"2024-09-10T19:01:52.613469+0000","last_peered":"2024-09-10T19:01:52.613469+0000","last_clean":"2024-09-10T19:01:52.613469+0000","last_became_active":"2024-09-10T19:01:44.344810+0000","last_became_peered":"2024-09-10T19:01:44.344810+0000","last_unstale":"2024-09-10T19:01:52.613469+0000","last_undegraded":"2024-09-10T19:01:52.613469+0000","last_fullsized":"2024-09-10T19:01:52.613469+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:54:04.769483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335157+0000","last_change":"2024-09-10T19:01:46.343343+0000","last_active":"2024-09-10T19:01:51.335157+0000","last_peered":"2024-09-10T19:01:51.335157+0000","last_clean":"2024-09-10T19:01:51.335157+0000","last_became_active":"2024-09-10T19:01:46.343035+0000","last_became_peered":"2024-09-10T19:01:46.343035+0000","last_unstale":"2024-09-10T19:01:51.335157+0000","last_undegraded":"2024-09-10T19:01:51.335157+0000","last_fullsized":"2024-09-10T19:01:51.335157+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:58:45.098178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"59'10","reported_seq":31,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330458+0000","last_change":"2024-09-10T19:01:42.322765+0000","last_active":"2024-09-10T19:01:51.330458+0000","last_peered":"2024-09-10T19:01:51.330458+0000","last_clean":"2024-09-10T19:01:51.330458+0000","last_became_active":"2024-09-10T19:01:42.322436+0000","last_became_peered":"2024-09-10T19:01:42.322436+0000","last_unstale":"2024-09-10T19:01:51.330458+0000","last_undegraded":"2024-09-10T19:01:51.330458+0000","last_fullsized":"2024-09-10T19:01:51.330458+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:09:55.001758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614549+0000","last_change":"2024-09-10T19:01:40.316538+0000","last_active":"2024-09-10T19:01:52.614549+0000","last_peered":"2024-09-10T19:01:52.614549+0000","last_clean":"2024-09-10T19:01:52.614549+0000","last_became_active":"2024-09-10T19:01:40.315880+0000","last_became_peered":"2024-09-10T19:01:40.315880+0000","last_unstale":"2024-09-10T19:01:52.614549+0000","last_undegraded":"2024-09-10T19:01:52.614549+0000","last_fullsized":"2024-09-10T19:01:52.614549+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:03:06.440224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"59'8","reported_seq":38,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:53.778679+0000","last_change":"2024-09-10T19:01:44.340940+0000","last_active":"2024-09-10T19:02:53.778679+0000","last_peered":"2024-09-10T19:02:53.778679+0000","last_clean":"2024-09-10T19:02:53.778679+0000","last_became_active":"2024-09-10T19:01:44.339818+0000","last_became_peered":"2024-09-10T19:01:44.339818+0000","last_unstale":"2024-09-10T19:02:53.778679+0000","last_undegraded":"2024-09-10T19:02:53.778679+0000","last_fullsized":"2024-09-10T19:02:53.778679+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:15:31.816876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330517+0000","last_change":"2024-09-10T19:01:46.346880+0000","last_active":"2024-09-10T19:01:51.330517+0000","last_peered":"2024-09-10T19:01:51.330517+0000","last_clean":"2024-09-10T19:01:51.330517+0000","last_became_active":"2024-09-10T19:01:46.346609+0000","last_became_peered":"2024-09-10T19:01:46.346609+0000","last_unstale":"2024-09-10T19:01:51.330517+0000","last_undegraded":"2024-09-10T19:01:51.330517+0000","last_fullsized":"2024-09-10T19:01:51.330517+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:14.026153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"59'15","reported_seq":41,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.615071+0000","last_change":"2024-09-10T19:01:42.330612+0000","last_active":"2024-09-10T19:01:52.615071+0000","last_peered":"2024-09-10T19:01:52.615071+0000","last_clean":"2024-09-10T19:01:52.615071+0000","last_became_active":"2024-09-10T19:01:42.330264+0000","last_became_peered":"2024-09-10T19:01:42.330264+0000","last_unstale":"2024-09-10T19:01:52.615071+0000","last_undegraded":"2024-09-10T19:01:52.615071+0000","last_fullsized":"2024-09-10T19:01:52.615071+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:22:26.401805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.613902+0000","last_change":"2024-09-10T19:01:40.315446+0000","last_active":"2024-09-10T19:01:52.613902+0000","last_peered":"2024-09-10T19:01:52.613902+0000","last_clean":"2024-09-10T19:01:52.613902+0000","last_became_active":"2024-09-10T19:01:40.315185+0000","last_became_peered":"2024-09-10T19:01:40.315185+0000","last_unstale":"2024-09-10T19:01:52.613902+0000","last_undegraded":"2024-09-10T19:01:52.613902+0000","last_fullsized":"2024-09-10T19:01:52.613902+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:12:52.822736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"59'8","reported_seq":38,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:53.779151+0000","last_change":"2024-09-10T19:01:44.350077+0000","last_active":"2024-09-10T19:02:53.779151+0000","last_peered":"2024-09-10T19:02:53.779151+0000","last_clean":"2024-09-10T19:02:53.779151+0000","last_became_active":"2024-09-10T19:01:44.349746+0000","last_became_peered":"2024-09-10T19:01:44.349746+0000","last_unstale":"2024-09-10T19:02:53.779151+0000","last_undegraded":"2024-09-10T19:02:53.779151+0000","last_fullsized":"2024-09-10T19:02:53.779151+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:59.302630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.347086+0000","last_change":"2024-09-10T19:01:46.347513+0000","last_active":"2024-09-10T19:01:52.347086+0000","last_peered":"2024-09-10T19:01:52.347086+0000","last_clean":"2024-09-10T19:01:52.347086+0000","last_became_active":"2024-09-10T19:01:46.347277+0000","last_became_peered":"2024-09-10T19:01:46.347277+0000","last_unstale":"2024-09-10T19:01:52.347086+0000","last_undegraded":"2024-09-10T19:01:52.347086+0000","last_fullsized":"2024-09-10T19:01:52.347086+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:32:53.136513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"59'11","reported_seq":35,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330627+0000","last_change":"2024-09-10T19:01:42.325306+0000","last_active":"2024-09-10T19:01:51.330627+0000","last_peered":"2024-09-10T19:01:51.330627+0000","last_clean":"2024-09-10T19:01:51.330627+0000","last_became_active":"2024-09-10T19:01:42.325044+0000","last_became_peered":"2024-09-10T19:01:42.325044+0000","last_unstale":"2024-09-10T19:01:51.330627+0000","last_undegraded":"2024-09-10T19:01:51.330627+0000","last_fullsized":"2024-09-10T19:01:51.330627+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:34:05.813571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.473878+0000","last_change":"2024-09-10T19:01:40.322115+0000","last_active":"2024-09-10T19:01:52.473878+0000","last_peered":"2024-09-10T19:01:52.473878+0000","last_clean":"2024-09-10T19:01:52.473878+0000","last_became_active":"2024-09-10T19:01:40.321510+0000","last_became_peered":"2024-09-10T19:01:40.321510+0000","last_unstale":"2024-09-10T19:01:52.473878+0000","last_undegraded":"2024-09-10T19:01:52.473878+0000","last_fullsized":"2024-09-10T19:01:52.473878+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:22:12.047819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346704+0000","last_change":"2024-09-10T19:01:44.344680+0000","last_active":"2024-09-10T19:01:52.346704+0000","last_peered":"2024-09-10T19:01:52.346704+0000","last_clean":"2024-09-10T19:01:52.346704+0000","last_became_active":"2024-09-10T19:01:44.344390+0000","last_became_peered":"2024-09-10T19:01:44.344390+0000","last_unstale":"2024-09-10T19:01:52.346704+0000","last_undegraded":"2024-09-10T19:01:52.346704+0000","last_fullsized":"2024-09-10T19:01:52.346704+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:25:17.312301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976932+0000","last_change":"2024-09-10T19:01:46.614346+0000","last_active":"2024-09-10T19:01:51.976932+0000","last_peered":"2024-09-10T19:01:51.976932+0000","last_clean":"2024-09-10T19:01:51.976932+0000","last_became_active":"2024-09-10T19:01:46.613951+0000","last_became_peered":"2024-09-10T19:01:46.613951+0000","last_unstale":"2024-09-10T19:01:51.976932+0000","last_undegraded":"2024-09-10T19:01:51.976932+0000","last_fullsized":"2024-09-10T19:01:51.976932+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:32:16.513970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"59'14","reported_seq":37,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330199+0000","last_change":"2024-09-10T19:01:42.326162+0000","last_active":"2024-09-10T19:01:51.330199+0000","last_peered":"2024-09-10T19:01:51.330199+0000","last_clean":"2024-09-10T19:01:51.330199+0000","last_became_active":"2024-09-10T19:01:42.325837+0000","last_became_peered":"2024-09-10T19:01:42.325837+0000","last_unstale":"2024-09-10T19:01:51.330199+0000","last_undegraded":"2024-09-10T19:01:51.330199+0000","last_fullsized":"2024-09-10T19:01:51.330199+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:15.681351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.609066+0000","last_change":"2024-09-10T19:01:40.313839+0000","last_active":"2024-09-10T19:01:52.609066+0000","last_peered":"2024-09-10T19:01:52.609066+0000","last_clean":"2024-09-10T19:01:52.609066+0000","last_became_active":"2024-09-10T19:01:40.313296+0000","last_became_peered":"2024-09-10T19:01:40.313296+0000","last_unstale":"2024-09-10T19:01:52.609066+0000","last_undegraded":"2024-09-10T19:01:52.609066+0000","last_fullsized":"2024-09-10T19:01:52.609066+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:44:00.974249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"59'8","reported_seq":38,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:53.778799+0000","last_change":"2024-09-10T19:01:44.349408+0000","last_active":"2024-09-10T19:02:53.778799+0000","last_peered":"2024-09-10T19:02:53.778799+0000","last_clean":"2024-09-10T19:02:53.778799+0000","last_became_active":"2024-09-10T19:01:44.349080+0000","last_became_peered":"2024-09-10T19:01:44.349080+0000","last_unstale":"2024-09-10T19:02:53.778799+0000","last_undegraded":"2024-09-10T19:02:53.778799+0000","last_fullsized":"2024-09-10T19:02:53.778799+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:56:31.359086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474222+0000","last_change":"2024-09-10T19:01:46.614432+0000","last_active":"2024-09-10T19:01:52.474222+0000","last_peered":"2024-09-10T19:01:52.474222+0000","last_clean":"2024-09-10T19:01:52.474222+0000","last_became_active":"2024-09-10T19:01:46.614179+0000","last_became_peered":"2024-09-10T19:01:46.614179+0000","last_unstale":"2024-09-10T19:01:52.474222+0000","last_undegraded":"2024-09-10T19:01:52.474222+0000","last_fullsized":"2024-09-10T19:01:52.474222+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:43:41.707412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"59'18","reported_seq":48,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976646+0000","last_change":"2024-09-10T19:01:42.381843+0000","last_active":"2024-09-10T19:01:51.976646+0000","last_peered":"2024-09-10T19:01:51.976646+0000","last_clean":"2024-09-10T19:01:51.976646+0000","last_became_active":"2024-09-10T19:01:42.381564+0000","last_became_peered":"2024-09-10T19:01:42.381564+0000","last_unstale":"2024-09-10T19:01:51.976646+0000","last_undegraded":"2024-09-10T19:01:51.976646+0000","last_fullsized":"2024-09-10T19:01:51.976646+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:53:40.137432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614351+0000","last_change":"2024-09-10T19:01:40.310806+0000","last_active":"2024-09-10T19:01:52.614351+0000","last_peered":"2024-09-10T19:01:52.614351+0000","last_clean":"2024-09-10T19:01:52.614351+0000","last_became_active":"2024-09-10T19:01:40.310526+0000","last_became_peered":"2024-09-10T19:01:40.310526+0000","last_unstale":"2024-09-10T19:01:52.614351+0000","last_undegraded":"2024-09-10T19:01:52.614351+0000","last_fullsized":"2024-09-10T19:01:52.614351+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:08:45.808788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330069+0000","last_change":"2024-09-10T19:01:44.344688+0000","last_active":"2024-09-10T19:01:51.330069+0000","last_peered":"2024-09-10T19:01:51.330069+0000","last_clean":"2024-09-10T19:01:51.330069+0000","last_became_active":"2024-09-10T19:01:44.344387+0000","last_became_peered":"2024-09-10T19:01:44.344387+0000","last_unstale":"2024-09-10T19:01:51.330069+0000","last_undegraded":"2024-09-10T19:01:51.330069+0000","last_fullsized":"2024-09-10T19:01:51.330069+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:31.388096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330246+0000","last_change":"2024-09-10T19:01:46.346430+0000","last_active":"2024-09-10T19:01:51.330246+0000","last_peered":"2024-09-10T19:01:51.330246+0000","last_clean":"2024-09-10T19:01:51.330246+0000","last_became_active":"2024-09-10T19:01:46.346135+0000","last_became_peered":"2024-09-10T19:01:46.346135+0000","last_unstale":"2024-09-10T19:01:51.330246+0000","last_undegraded":"2024-09-10T19:01:51.330246+0000","last_fullsized":"2024-09-10T19:01:51.330246+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:40.356164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"59'19","reported_seq":52,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336397+0000","last_change":"2024-09-10T19:01:42.379107+0000","last_active":"2024-09-10T19:01:51.336397+0000","last_peered":"2024-09-10T19:01:51.336397+0000","last_clean":"2024-09-10T19:01:51.336397+0000","last_became_active":"2024-09-10T19:01:42.378807+0000","last_became_peered":"2024-09-10T19:01:42.378807+0000","last_unstale":"2024-09-10T19:01:51.336397+0000","last_undegraded":"2024-09-10T19:01:51.336397+0000","last_fullsized":"2024-09-10T19:01:51.336397+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:21:08.942965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474690+0000","last_change":"2024-09-10T19:01:40.318315+0000","last_active":"2024-09-10T19:01:52.474690+0000","last_peered":"2024-09-10T19:01:52.474690+0000","last_clean":"2024-09-10T19:01:52.474690+0000","last_became_active":"2024-09-10T19:01:40.318059+0000","last_became_peered":"2024-09-10T19:01:40.318059+0000","last_unstale":"2024-09-10T19:01:52.474690+0000","last_undegraded":"2024-09-10T19:01:52.474690+0000","last_fullsized":"2024-09-10T19:01:52.474690+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:24:29.750728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.607282+0000","last_change":"2024-09-10T19:01:44.345624+0000","last_active":"2024-09-10T19:01:52.607282+0000","last_peered":"2024-09-10T19:01:52.607282+0000","last_clean":"2024-09-10T19:01:52.607282+0000","last_became_active":"2024-09-10T19:01:44.345282+0000","last_became_peered":"2024-09-10T19:01:44.345282+0000","last_unstale":"2024-09-10T19:01:52.607282+0000","last_undegraded":"2024-09-10T19:01:52.607282+0000","last_fullsized":"2024-09-10T19:01:52.607282+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:45.825160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.613698+0000","last_change":"2024-09-10T19:01:46.614413+0000","last_active":"2024-09-10T19:01:52.613698+0000","last_peered":"2024-09-10T19:01:52.613698+0000","last_clean":"2024-09-10T19:01:52.613698+0000","last_became_active":"2024-09-10T19:01:46.613982+0000","last_became_peered":"2024-09-10T19:01:46.613982+0000","last_unstale":"2024-09-10T19:01:52.613698+0000","last_undegraded":"2024-09-10T19:01:52.613698+0000","last_fullsized":"2024-09-10T19:01:52.613698+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:16:42.295482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"59'10","reported_seq":31,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614429+0000","last_change":"2024-09-10T19:01:42.326777+0000","last_active":"2024-09-10T19:01:52.614429+0000","last_peered":"2024-09-10T19:01:52.614429+0000","last_clean":"2024-09-10T19:01:52.614429+0000","last_became_active":"2024-09-10T19:01:42.326245+0000","last_became_peered":"2024-09-10T19:01:42.326245+0000","last_unstale":"2024-09-10T19:01:52.614429+0000","last_undegraded":"2024-09-10T19:01:52.614429+0000","last_fullsized":"2024-09-10T19:01:52.614429+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:31:45.800699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614981+0000","last_change":"2024-09-10T19:01:40.317340+0000","last_active":"2024-09-10T19:01:52.614981+0000","last_peered":"2024-09-10T19:01:52.614981+0000","last_clean":"2024-09-10T19:01:52.614981+0000","last_became_active":"2024-09-10T19:01:40.317088+0000","last_became_peered":"2024-09-10T19:01:40.317088+0000","last_unstale":"2024-09-10T19:01:52.614981+0000","last_undegraded":"2024-09-10T19:01:52.614981+0000","last_fullsized":"2024-09-10T19:01:52.614981+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:46:56.608265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"59'8","reported_seq":41,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:53.778533+0000","last_change":"2024-09-10T19:01:44.343307+0000","last_active":"2024-09-10T19:02:53.778533+0000","last_peered":"2024-09-10T19:02:53.778533+0000","last_clean":"2024-09-10T19:02:53.778533+0000","last_became_active":"2024-09-10T19:01:44.342984+0000","last_became_peered":"2024-09-10T19:01:44.342984+0000","last_unstale":"2024-09-10T19:02:53.778533+0000","last_undegraded":"2024-09-10T19:02:53.778533+0000","last_fullsized":"2024-09-10T19:02:53.778533+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:51:22.024642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335605+0000","last_change":"2024-09-10T19:01:46.346173+0000","last_active":"2024-09-10T19:01:51.335605+0000","last_peered":"2024-09-10T19:01:51.335605+0000","last_clean":"2024-09-10T19:01:51.335605+0000","last_became_active":"2024-09-10T19:01:46.345815+0000","last_became_peered":"2024-09-10T19:01:46.345815+0000","last_unstale":"2024-09-10T19:01:51.335605+0000","last_undegraded":"2024-09-10T19:01:51.335605+0000","last_fullsized":"2024-09-10T19:01:51.335605+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:59:56.757802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"59'28","reported_seq":80,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:53.779023+0000","last_change":"2024-09-10T19:01:42.331262+0000","last_active":"2024-09-10T19:02:53.779023+0000","last_peered":"2024-09-10T19:02:53.779023+0000","last_clean":"2024-09-10T19:02:53.779023+0000","last_became_active":"2024-09-10T19:01:42.330911+0000","last_became_peered":"2024-09-10T19:01:42.330911+0000","last_unstale":"2024-09-10T19:02:53.779023+0000","last_undegraded":"2024-09-10T19:02:53.779023+0000","last_fullsized":"2024-09-10T19:02:53.779023+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:08:58.322841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346938+0000","last_change":"2024-09-10T19:01:40.315419+0000","last_active":"2024-09-10T19:01:52.346938+0000","last_peered":"2024-09-10T19:01:52.346938+0000","last_clean":"2024-09-10T19:01:52.346938+0000","last_became_active":"2024-09-10T19:01:40.315161+0000","last_became_peered":"2024-09-10T19:01:40.315161+0000","last_unstale":"2024-09-10T19:01:52.346938+0000","last_undegraded":"2024-09-10T19:01:52.346938+0000","last_fullsized":"2024-09-10T19:01:52.346938+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:51:22.926469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336283+0000","last_change":"2024-09-10T19:01:44.344639+0000","last_active":"2024-09-10T19:01:51.336283+0000","last_peered":"2024-09-10T19:01:51.336283+0000","last_clean":"2024-09-10T19:01:51.336283+0000","last_became_active":"2024-09-10T19:01:44.344352+0000","last_became_peered":"2024-09-10T19:01:44.344352+0000","last_unstale":"2024-09-10T19:01:51.336283+0000","last_undegraded":"2024-09-10T19:01:51.336283+0000","last_fullsized":"2024-09-10T19:01:51.336283+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:26:08.244949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977330+0000","last_change":"2024-09-10T19:01:46.347861+0000","last_active":"2024-09-10T19:01:51.977330+0000","last_peered":"2024-09-10T19:01:51.977330+0000","last_clean":"2024-09-10T19:01:51.977330+0000","last_became_active":"2024-09-10T19:01:46.345916+0000","last_became_peered":"2024-09-10T19:01:46.345916+0000","last_unstale":"2024-09-10T19:01:51.977330+0000","last_undegraded":"2024-09-10T19:01:51.977330+0000","last_fullsized":"2024-09-10T19:01:51.977330+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:57.183548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"59'13","reported_seq":43,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.615102+0000","last_change":"2024-09-10T19:01:42.330224+0000","last_active":"2024-09-10T19:01:52.615102+0000","last_peered":"2024-09-10T19:01:52.615102+0000","last_clean":"2024-09-10T19:01:52.615102+0000","last_became_active":"2024-09-10T19:01:42.329948+0000","last_became_peered":"2024-09-10T19:01:42.329948+0000","last_unstale":"2024-09-10T19:01:52.615102+0000","last_undegraded":"2024-09-10T19:01:52.615102+0000","last_fullsized":"2024-09-10T19:01:52.615102+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:08.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"60'5","reported_seq":102,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:03:01.751409+0000","last_change":"2024-09-10T19:01:44.339700+0000","last_active":"2024-09-10T19:03:01.751409+0000","last_peered":"2024-09-10T19:03:01.751409+0000","last_clean":"2024-09-10T19:03:01.751409+0000","last_became_active":"2024-09-10T19:01:40.310716+0000","last_became_peered":"2024-09-10T19:01:40.310716+0000","last_unstale":"2024-09-10T19:03:01.751409+0000","last_undegraded":"2024-09-10T19:03:01.751409+0000","last_fullsized":"2024-09-10T19:03:01.751409+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:52:23.632309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033453020000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.334861+0000","last_change":"2024-09-10T19:01:40.310863+0000","last_active":"2024-09-10T19:01:51.334861+0000","last_peered":"2024-09-10T19:01:51.334861+0000","last_clean":"2024-09-10T19:01:51.334861+0000","last_became_active":"2024-09-10T19:01:40.310551+0000","last_became_peered":"2024-09-10T19:01:40.310551+0000","last_unstale":"2024-09-10T19:01:51.334861+0000","last_undegraded":"2024-09-10T19:01:51.334861+0000","last_fullsized":"2024-09-10T19:01:51.334861+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:51:34.081310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335227+0000","last_change":"2024-09-10T19:01:44.341197+0000","last_active":"2024-09-10T19:01:51.335227+0000","last_peered":"2024-09-10T19:01:51.335227+0000","last_clean":"2024-09-10T19:01:51.335227+0000","last_became_active":"2024-09-10T19:01:44.340361+0000","last_became_peered":"2024-09-10T19:01:44.340361+0000","last_unstale":"2024-09-10T19:01:51.335227+0000","last_undegraded":"2024-09-10T19:01:51.335227+0000","last_fullsized":"2024-09-10T19:01:51.335227+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:38:25.636722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474539+0000","last_change":"2024-09-10T19:01:46.614318+0000","last_active":"2024-09-10T19:01:52.474539+0000","last_peered":"2024-09-10T19:01:52.474539+0000","last_clean":"2024-09-10T19:01:52.474539+0000","last_became_active":"2024-09-10T19:01:46.614022+0000","last_became_peered":"2024-09-10T19:01:46.614022+0000","last_unstale":"2024-09-10T19:01:52.474539+0000","last_undegraded":"2024-09-10T19:01:52.474539+0000","last_fullsized":"2024-09-10T19:01:52.474539+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:40:01.399069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"59'12","reported_seq":34,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336634+0000","last_change":"2024-09-10T19:01:42.328339+0000","last_active":"2024-09-10T19:01:51.336634+0000","last_peered":"2024-09-10T19:01:51.336634+0000","last_clean":"2024-09-10T19:01:51.336634+0000","last_became_active":"2024-09-10T19:01:42.328027+0000","last_became_peered":"2024-09-10T19:01:42.328027+0000","last_unstale":"2024-09-10T19:01:51.336634+0000","last_undegraded":"2024-09-10T19:01:51.336634+0000","last_fullsized":"2024-09-10T19:01:51.336634+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:11:20.315951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"54'1","reported_seq":24,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336081+0000","last_change":"2024-09-10T19:01:44.331881+0000","last_active":"2024-09-10T19:01:51.336081+0000","last_peered":"2024-09-10T19:01:51.336081+0000","last_clean":"2024-09-10T19:01:51.336081+0000","last_became_active":"2024-09-10T19:01:40.310783+0000","last_became_peered":"2024-09-10T19:01:40.310783+0000","last_unstale":"2024-09-10T19:01:51.336081+0000","last_undegraded":"2024-09-10T19:01:51.336081+0000","last_fullsized":"2024-09-10T19:01:51.336081+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:16:45.275300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043770099999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474452+0000","last_change":"2024-09-10T19:01:40.321999+0000","last_active":"2024-09-10T19:01:52.474452+0000","last_peered":"2024-09-10T19:01:52.474452+0000","last_clean":"2024-09-10T19:01:52.474452+0000","last_became_active":"2024-09-10T19:01:40.321349+0000","last_became_peered":"2024-09-10T19:01:40.321349+0000","last_unstale":"2024-09-10T19:01:52.474452+0000","last_undegraded":"2024-09-10T19:01:52.474452+0000","last_fullsized":"2024-09-10T19:01:52.474452+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:46:19.376593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346684+0000","last_change":"2024-09-10T19:01:44.344941+0000","last_active":"2024-09-10T19:01:52.346684+0000","last_peered":"2024-09-10T19:01:52.346684+0000","last_clean":"2024-09-10T19:01:52.346684+0000","last_became_active":"2024-09-10T19:01:44.344615+0000","last_became_peered":"2024-09-10T19:01:44.344615+0000","last_unstale":"2024-09-10T19:01:52.346684+0000","last_undegraded":"2024-09-10T19:01:52.346684+0000","last_fullsized":"2024-09-10T19:01:52.346684+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:25:34.432176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614751+0000","last_change":"2024-09-10T19:01:46.346738+0000","last_active":"2024-09-10T19:01:52.614751+0000","last_peered":"2024-09-10T19:01:52.614751+0000","last_clean":"2024-09-10T19:01:52.614751+0000","last_became_active":"2024-09-10T19:01:46.346475+0000","last_became_peered":"2024-09-10T19:01:46.346475+0000","last_unstale":"2024-09-10T19:01:52.614751+0000","last_undegraded":"2024-09-10T19:01:52.614751+0000","last_fullsized":"2024-09-10T19:01:52.614751+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:32:21.180179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"59'14","reported_seq":38,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.608369+0000","last_change":"2024-09-10T19:01:42.380619+0000","last_active":"2024-09-10T19:01:52.608369+0000","last_peered":"2024-09-10T19:01:52.608369+0000","last_clean":"2024-09-10T19:01:52.608369+0000","last_became_active":"2024-09-10T19:01:42.380306+0000","last_became_peered":"2024-09-10T19:01:42.380306+0000","last_unstale":"2024-09-10T19:01:52.608369+0000","last_undegraded":"2024-09-10T19:01:52.608369+0000","last_fullsized":"2024-09-10T19:01:52.608369+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:54:18.570288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"56'2","reported_seq":25,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977002+0000","last_change":"2024-09-10T19:01:44.339090+0000","last_active":"2024-09-10T19:01:51.977002+0000","last_peered":"2024-09-10T19:01:51.977002+0000","last_clean":"2024-09-10T19:01:51.977002+0000","last_became_active":"2024-09-10T19:01:40.313472+0000","last_became_peered":"2024-09-10T19:01:40.313472+0000","last_unstale":"2024-09-10T19:01:51.977002+0000","last_undegraded":"2024-09-10T19:01:51.977002+0000","last_fullsized":"2024-09-10T19:01:51.977002+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:52:25.173129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019757860000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346883+0000","last_change":"2024-09-10T19:01:40.319837+0000","last_active":"2024-09-10T19:01:52.346883+0000","last_peered":"2024-09-10T19:01:52.346883+0000","last_clean":"2024-09-10T19:01:52.346883+0000","last_became_active":"2024-09-10T19:01:40.319316+0000","last_became_peered":"2024-09-10T19:01:40.319316+0000","last_unstale":"2024-09-10T19:01:52.346883+0000","last_undegraded":"2024-09-10T19:01:52.346883+0000","last_fullsized":"2024-09-10T19:01:52.346883+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:48:30.471262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"61'79","reported_seq":52,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:19.076563+0000","last_change":"2024-09-10T19:01:26.169270+0000","last_active":"2024-09-10T19:02:19.076563+0000","last_peered":"2024-09-10T19:02:19.076563+0000","last_clean":"2024-09-10T19:02:19.076563+0000","last_became_active":"2024-09-10T19:01:26.161939+0000","last_became_peered":"2024-09-10T19:01:26.161939+0000","last_unstale":"2024-09-10T19:02:19.076563+0000","last_undegraded":"2024-09-10T19:02:19.076563+0000","last_fullsized":"2024-09-10T19:02:19.076563+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:30.525956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:30.525956+0000","last_clean_scrub_stamp":"2024-09-10T18:59:30.525956+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:57:46.815060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.473602+0000","last_change":"2024-09-10T19:01:44.342189+0000","last_active":"2024-09-10T19:01:52.473602+0000","last_peered":"2024-09-10T19:01:52.473602+0000","last_clean":"2024-09-10T19:01:52.473602+0000","last_became_active":"2024-09-10T19:01:44.341900+0000","last_became_peered":"2024-09-10T19:01:44.341900+0000","last_unstale":"2024-09-10T19:01:52.473602+0000","last_undegraded":"2024-09-10T19:01:52.473602+0000","last_fullsized":"2024-09-10T19:01:52.473602+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:26:39.302809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.347209+0000","last_change":"2024-09-10T19:01:46.350066+0000","last_active":"2024-09-10T19:01:52.347209+0000","last_peered":"2024-09-10T19:01:52.347209+0000","last_clean":"2024-09-10T19:01:52.347209+0000","last_became_active":"2024-09-10T19:01:46.349776+0000","last_became_peered":"2024-09-10T19:01:46.349776+0000","last_unstale":"2024-09-10T19:01:52.347209+0000","last_undegraded":"2024-09-10T19:01:52.347209+0000","last_fullsized":"2024-09-10T19:01:52.347209+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:27.446823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"59'19","reported_seq":47,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.607674+0000","last_change":"2024-09-10T19:01:42.379603+0000","last_active":"2024-09-10T19:01:52.607674+0000","last_peered":"2024-09-10T19:01:52.607674+0000","last_clean":"2024-09-10T19:01:52.607674+0000","last_became_active":"2024-09-10T19:01:42.378718+0000","last_became_peered":"2024-09-10T19:01:42.378718+0000","last_unstale":"2024-09-10T19:01:52.607674+0000","last_undegraded":"2024-09-10T19:01:52.607674+0000","last_fullsized":"2024-09-10T19:01:52.607674+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:24:21.723058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335307+0000","last_change":"2024-09-10T19:01:40.312962+0000","last_active":"2024-09-10T19:01:51.335307+0000","last_peered":"2024-09-10T19:01:51.335307+0000","last_clean":"2024-09-10T19:01:51.335307+0000","last_became_active":"2024-09-10T19:01:40.312720+0000","last_became_peered":"2024-09-10T19:01:40.312720+0000","last_unstale":"2024-09-10T19:01:51.335307+0000","last_undegraded":"2024-09-10T19:01:51.335307+0000","last_fullsized":"2024-09-10T19:01:51.335307+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:00:08.645352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.334746+0000","last_change":"2024-09-10T19:01:44.340037+0000","last_active":"2024-09-10T19:01:51.334746+0000","last_peered":"2024-09-10T19:01:51.334746+0000","last_clean":"2024-09-10T19:01:51.334746+0000","last_became_active":"2024-09-10T19:01:44.339699+0000","last_became_peered":"2024-09-10T19:01:44.339699+0000","last_unstale":"2024-09-10T19:01:51.334746+0000","last_undegraded":"2024-09-10T19:01:51.334746+0000","last_fullsized":"2024-09-10T19:01:51.334746+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:16:51.938721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.473990+0000","last_change":"2024-09-10T19:01:46.347613+0000","last_active":"2024-09-10T19:01:52.473990+0000","last_peered":"2024-09-10T19:01:52.473990+0000","last_clean":"2024-09-10T19:01:52.473990+0000","last_became_active":"2024-09-10T19:01:46.347345+0000","last_became_peered":"2024-09-10T19:01:46.347345+0000","last_unstale":"2024-09-10T19:01:52.473990+0000","last_undegraded":"2024-09-10T19:01:52.473990+0000","last_fullsized":"2024-09-10T19:01:52.473990+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:27:43.490758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"59'9","reported_seq":32,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336296+0000","last_change":"2024-09-10T19:01:42.325200+0000","last_active":"2024-09-10T19:01:51.336296+0000","last_peered":"2024-09-10T19:01:51.336296+0000","last_clean":"2024-09-10T19:01:51.336296+0000","last_became_active":"2024-09-10T19:01:42.324853+0000","last_became_peered":"2024-09-10T19:01:42.324853+0000","last_unstale":"2024-09-10T19:01:51.336296+0000","last_undegraded":"2024-09-10T19:01:51.336296+0000","last_fullsized":"2024-09-10T19:01:51.336296+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:46:53.265923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614900+0000","last_change":"2024-09-10T19:01:40.308532+0000","last_active":"2024-09-10T19:01:52.614900+0000","last_peered":"2024-09-10T19:01:52.614900+0000","last_clean":"2024-09-10T19:01:52.614900+0000","last_became_active":"2024-09-10T19:01:40.308261+0000","last_became_peered":"2024-09-10T19:01:40.308261+0000","last_unstale":"2024-09-10T19:01:52.614900+0000","last_undegraded":"2024-09-10T19:01:52.614900+0000","last_fullsized":"2024-09-10T19:01:52.614900+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:53:24.762541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.334722+0000","last_change":"2024-09-10T19:01:44.343461+0000","last_active":"2024-09-10T19:01:51.334722+0000","last_peered":"2024-09-10T19:01:51.334722+0000","last_clean":"2024-09-10T19:01:51.334722+0000","last_became_active":"2024-09-10T19:01:44.342749+0000","last_became_peered":"2024-09-10T19:01:44.342749+0000","last_unstale":"2024-09-10T19:01:51.334722+0000","last_undegraded":"2024-09-10T19:01:51.334722+0000","last_fullsized":"2024-09-10T19:01:51.334722+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:53.728630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335889+0000","last_change":"2024-09-10T19:01:46.346711+0000","last_active":"2024-09-10T19:01:51.335889+0000","last_peered":"2024-09-10T19:01:51.335889+0000","last_clean":"2024-09-10T19:01:51.335889+0000","last_became_active":"2024-09-10T19:01:46.346422+0000","last_became_peered":"2024-09-10T19:01:46.346422+0000","last_unstale":"2024-09-10T19:01:51.335889+0000","last_undegraded":"2024-09-10T19:01:51.335889+0000","last_fullsized":"2024-09-10T19:01:51.335889+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:21.410686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"59'15","reported_seq":41,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346592+0000","last_change":"2024-09-10T19:01:42.379088+0000","last_active":"2024-09-10T19:01:52.346592+0000","last_peered":"2024-09-10T19:01:52.346592+0000","last_clean":"2024-09-10T19:01:52.346592+0000","last_became_active":"2024-09-10T19:01:42.378763+0000","last_became_peered":"2024-09-10T19:01:42.378763+0000","last_unstale":"2024-09-10T19:01:52.346592+0000","last_undegraded":"2024-09-10T19:01:52.346592+0000","last_fullsized":"2024-09-10T19:01:52.346592+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:49:33.777985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335296+0000","last_change":"2024-09-10T19:01:40.313052+0000","last_active":"2024-09-10T19:01:51.335296+0000","last_peered":"2024-09-10T19:01:51.335296+0000","last_clean":"2024-09-10T19:01:51.335296+0000","last_became_active":"2024-09-10T19:01:40.312787+0000","last_became_peered":"2024-09-10T19:01:40.312787+0000","last_unstale":"2024-09-10T19:01:51.335296+0000","last_undegraded":"2024-09-10T19:01:51.335296+0000","last_fullsized":"2024-09-10T19:01:51.335296+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:11:55.269886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"59'8","reported_seq":38,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:53.778751+0000","last_change":"2024-09-10T19:01:44.344555+0000","last_active":"2024-09-10T19:02:53.778751+0000","last_peered":"2024-09-10T19:02:53.778751+0000","last_clean":"2024-09-10T19:02:53.778751+0000","last_became_active":"2024-09-10T19:01:44.344303+0000","last_became_peered":"2024-09-10T19:01:44.344303+0000","last_unstale":"2024-09-10T19:02:53.778751+0000","last_undegraded":"2024-09-10T19:02:53.778751+0000","last_fullsized":"2024-09-10T19:02:53.778751+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:04.681831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346648+0000","last_change":"2024-09-10T19:01:46.615644+0000","last_active":"2024-09-10T19:01:52.346648+0000","last_peered":"2024-09-10T19:01:52.346648+0000","last_clean":"2024-09-10T19:01:52.346648+0000","last_became_active":"2024-09-10T19:01:46.614542+0000","last_became_peered":"2024-09-10T19:01:46.614542+0000","last_unstale":"2024-09-10T19:01:52.346648+0000","last_undegraded":"2024-09-10T19:01:52.346648+0000","last_fullsized":"2024-09-10T19:01:52.346648+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:53:22.911167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"59'12","reported_seq":39,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330103+0000","last_change":"2024-09-10T19:01:42.327921+0000","last_active":"2024-09-10T19:01:51.330103+0000","last_peered":"2024-09-10T19:01:51.330103+0000","last_clean":"2024-09-10T19:01:51.330103+0000","last_became_active":"2024-09-10T19:01:42.327569+0000","last_became_peered":"2024-09-10T19:01:42.327569+0000","last_unstale":"2024-09-10T19:01:51.330103+0000","last_undegraded":"2024-09-10T19:01:51.330103+0000","last_fullsized":"2024-09-10T19:01:51.330103+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:30.416835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"52'2","reported_seq":35,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330681+0000","last_change":"2024-09-10T19:01:40.313889+0000","last_active":"2024-09-10T19:01:51.330681+0000","last_peered":"2024-09-10T19:01:51.330681+0000","last_clean":"2024-09-10T19:01:51.330681+0000","last_became_active":"2024-09-10T19:01:40.313577+0000","last_became_peered":"2024-09-10T19:01:40.313577+0000","last_unstale":"2024-09-10T19:01:51.330681+0000","last_undegraded":"2024-09-10T19:01:51.330681+0000","last_fullsized":"2024-09-10T19:01:51.330681+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:28:09.705498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.334753+0000","last_change":"2024-09-10T19:01:44.340996+0000","last_active":"2024-09-10T19:01:51.334753+0000","last_peered":"2024-09-10T19:01:51.334753+0000","last_clean":"2024-09-10T19:01:51.334753+0000","last_became_active":"2024-09-10T19:01:44.340642+0000","last_became_peered":"2024-09-10T19:01:44.340642+0000","last_unstale":"2024-09-10T19:01:51.334753+0000","last_undegraded":"2024-09-10T19:01:51.334753+0000","last_fullsized":"2024-09-10T19:01:51.334753+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:58:27.613052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976778+0000","last_change":"2024-09-10T19:01:46.347759+0000","last_active":"2024-09-10T19:01:51.976778+0000","last_peered":"2024-09-10T19:01:51.976778+0000","last_clean":"2024-09-10T19:01:51.976778+0000","last_became_active":"2024-09-10T19:01:46.345502+0000","last_became_peered":"2024-09-10T19:01:46.345502+0000","last_unstale":"2024-09-10T19:01:51.976778+0000","last_undegraded":"2024-09-10T19:01:51.976778+0000","last_fullsized":"2024-09-10T19:01:51.976778+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:31.584396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"59'9","reported_seq":32,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336578+0000","last_change":"2024-09-10T19:01:42.379173+0000","last_active":"2024-09-10T19:01:51.336578+0000","last_peered":"2024-09-10T19:01:51.336578+0000","last_clean":"2024-09-10T19:01:51.336578+0000","last_became_active":"2024-09-10T19:01:42.378690+0000","last_became_peered":"2024-09-10T19:01:42.378690+0000","last_unstale":"2024-09-10T19:01:51.336578+0000","last_undegraded":"2024-09-10T19:01:51.336578+0000","last_fullsized":"2024-09-10T19:01:51.336578+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:49:32.120315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"52'1","reported_seq":27,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.334862+0000","last_change":"2024-09-10T19:01:40.311489+0000","last_active":"2024-09-10T19:01:51.334862+0000","last_peered":"2024-09-10T19:01:51.334862+0000","last_clean":"2024-09-10T19:01:51.334862+0000","last_became_active":"2024-09-10T19:01:40.311214+0000","last_became_peered":"2024-09-10T19:01:40.311214+0000","last_unstale":"2024-09-10T19:01:51.334862+0000","last_undegraded":"2024-09-10T19:01:51.334862+0000","last_fullsized":"2024-09-10T19:01:51.334862+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:15:11.232578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977495+0000","last_change":"2024-09-10T19:01:44.347275+0000","last_active":"2024-09-10T19:01:51.977495+0000","last_peered":"2024-09-10T19:01:51.977495+0000","last_clean":"2024-09-10T19:01:51.977495+0000","last_became_active":"2024-09-10T19:01:44.346984+0000","last_became_peered":"2024-09-10T19:01:44.346984+0000","last_unstale":"2024-09-10T19:01:51.977495+0000","last_undegraded":"2024-09-10T19:01:51.977495+0000","last_fullsized":"2024-09-10T19:01:51.977495+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:34:01.997058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335142+0000","last_change":"2024-09-10T19:01:46.346649+0000","last_active":"2024-09-10T19:01:51.335142+0000","last_peered":"2024-09-10T19:01:51.335142+0000","last_clean":"2024-09-10T19:01:51.335142+0000","last_became_active":"2024-09-10T19:01:46.346381+0000","last_became_peered":"2024-09-10T19:01:46.346381+0000","last_unstale":"2024-09-10T19:01:51.335142+0000","last_undegraded":"2024-09-10T19:01:51.335142+0000","last_fullsized":"2024-09-10T19:01:51.335142+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:24:14.210943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.607750+0000","last_change":"2024-09-10T19:01:40.312738+0000","last_active":"2024-09-10T19:01:52.607750+0000","last_peered":"2024-09-10T19:01:52.607750+0000","last_clean":"2024-09-10T19:01:52.607750+0000","last_became_active":"2024-09-10T19:01:40.312196+0000","last_became_peered":"2024-09-10T19:01:40.312196+0000","last_unstale":"2024-09-10T19:01:52.607750+0000","last_undegraded":"2024-09-10T19:01:52.607750+0000","last_fullsized":"2024-09-10T19:01:52.607750+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:18:29.748034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"59'6","reported_seq":25,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977548+0000","last_change":"2024-09-10T19:01:42.378946+0000","last_active":"2024-09-10T19:01:51.977548+0000","last_peered":"2024-09-10T19:01:51.977548+0000","last_clean":"2024-09-10T19:01:51.977548+0000","last_became_active":"2024-09-10T19:01:42.378693+0000","last_became_peered":"2024-09-10T19:01:42.378693+0000","last_unstale":"2024-09-10T19:01:51.977548+0000","last_undegraded":"2024-09-10T19:01:51.977548+0000","last_fullsized":"2024-09-10T19:01:51.977548+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:51:36.571214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346467+0000","last_change":"2024-09-10T19:01:44.343221+0000","last_active":"2024-09-10T19:01:52.346467+0000","last_peered":"2024-09-10T19:01:52.346467+0000","last_clean":"2024-09-10T19:01:52.346467+0000","last_became_active":"2024-09-10T19:01:44.342869+0000","last_became_peered":"2024-09-10T19:01:44.342869+0000","last_unstale":"2024-09-10T19:01:52.346467+0000","last_undegraded":"2024-09-10T19:01:52.346467+0000","last_fullsized":"2024-09-10T19:01:52.346467+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:38:26.130461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474546+0000","last_change":"2024-09-10T19:01:46.614487+0000","last_active":"2024-09-10T19:01:52.474546+0000","last_peered":"2024-09-10T19:01:52.474546+0000","last_clean":"2024-09-10T19:01:52.474546+0000","last_became_active":"2024-09-10T19:01:46.614229+0000","last_became_peered":"2024-09-10T19:01:46.614229+0000","last_unstale":"2024-09-10T19:01:52.474546+0000","last_undegraded":"2024-09-10T19:01:52.474546+0000","last_fullsized":"2024-09-10T19:01:52.474546+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:22.904465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"59'10","reported_seq":31,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977219+0000","last_change":"2024-09-10T19:01:42.378894+0000","last_active":"2024-09-10T19:01:51.977219+0000","last_peered":"2024-09-10T19:01:51.977219+0000","last_clean":"2024-09-10T19:01:51.977219+0000","last_became_active":"2024-09-10T19:01:42.378618+0000","last_became_peered":"2024-09-10T19:01:42.378618+0000","last_unstale":"2024-09-10T19:01:51.977219+0000","last_undegraded":"2024-09-10T19:01:51.977219+0000","last_fullsized":"2024-09-10T19:01:51.977219+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:52:35.762940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346955+0000","last_change":"2024-09-10T19:01:40.319717+0000","last_active":"2024-09-10T19:01:52.346955+0000","last_peered":"2024-09-10T19:01:52.346955+0000","last_clean":"2024-09-10T19:01:52.346955+0000","last_became_active":"2024-09-10T19:01:40.319447+0000","last_became_peered":"2024-09-10T19:01:40.319447+0000","last_unstale":"2024-09-10T19:01:52.346955+0000","last_undegraded":"2024-09-10T19:01:52.346955+0000","last_fullsized":"2024-09-10T19:01:52.346955+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:55.702268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"59'8","reported_seq":38,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:53.778655+0000","last_change":"2024-09-10T19:01:44.344270+0000","last_active":"2024-09-10T19:02:53.778655+0000","last_peered":"2024-09-10T19:02:53.778655+0000","last_clean":"2024-09-10T19:02:53.778655+0000","last_became_active":"2024-09-10T19:01:44.344000+0000","last_became_peered":"2024-09-10T19:01:44.344000+0000","last_unstale":"2024-09-10T19:02:53.778655+0000","last_undegraded":"2024-09-10T19:02:53.778655+0000","last_fullsized":"2024-09-10T19:02:53.778655+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:04:05.225998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"59'1","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336363+0000","last_change":"2024-09-10T19:01:46.347577+0000","last_active":"2024-09-10T19:01:51.336363+0000","last_peered":"2024-09-10T19:01:51.336363+0000","last_clean":"2024-09-10T19:01:51.336363+0000","last_became_active":"2024-09-10T19:01:46.347293+0000","last_became_peered":"2024-09-10T19:01:46.347293+0000","last_unstale":"2024-09-10T19:01:51.336363+0000","last_undegraded":"2024-09-10T19:01:51.336363+0000","last_fullsized":"2024-09-10T19:01:51.336363+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:12:23.785867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"59'9","reported_seq":32,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.347497+0000","last_change":"2024-09-10T19:01:42.380408+0000","last_active":"2024-09-10T19:01:52.347497+0000","last_peered":"2024-09-10T19:01:52.347497+0000","last_clean":"2024-09-10T19:01:52.347497+0000","last_became_active":"2024-09-10T19:01:42.380069+0000","last_became_peered":"2024-09-10T19:01:42.380069+0000","last_unstale":"2024-09-10T19:01:52.347497+0000","last_undegraded":"2024-09-10T19:01:52.347497+0000","last_fullsized":"2024-09-10T19:01:52.347497+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:55:22.417952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336538+0000","last_change":"2024-09-10T19:01:40.315253+0000","last_active":"2024-09-10T19:01:51.336538+0000","last_peered":"2024-09-10T19:01:51.336538+0000","last_clean":"2024-09-10T19:01:51.336538+0000","last_became_active":"2024-09-10T19:01:40.314605+0000","last_became_peered":"2024-09-10T19:01:40.314605+0000","last_unstale":"2024-09-10T19:01:51.336538+0000","last_undegraded":"2024-09-10T19:01:51.336538+0000","last_fullsized":"2024-09-10T19:01:51.336538+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:53:30.768785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"59'8","reported_seq":38,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:53.778575+0000","last_change":"2024-09-10T19:01:44.348398+0000","last_active":"2024-09-10T19:02:53.778575+0000","last_peered":"2024-09-10T19:02:53.778575+0000","last_clean":"2024-09-10T19:02:53.778575+0000","last_became_active":"2024-09-10T19:01:44.347356+0000","last_became_peered":"2024-09-10T19:01:44.347356+0000","last_unstale":"2024-09-10T19:02:53.778575+0000","last_undegraded":"2024-09-10T19:02:53.778575+0000","last_fullsized":"2024-09-10T19:02:53.778575+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:28:30.165391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330510+0000","last_change":"2024-09-10T19:01:46.348010+0000","last_active":"2024-09-10T19:01:51.330510+0000","last_peered":"2024-09-10T19:01:51.330510+0000","last_clean":"2024-09-10T19:01:51.330510+0000","last_became_active":"2024-09-10T19:01:46.346937+0000","last_became_peered":"2024-09-10T19:01:46.346937+0000","last_unstale":"2024-09-10T19:01:51.330510+0000","last_undegraded":"2024-09-10T19:01:51.330510+0000","last_fullsized":"2024-09-10T19:01:51.330510+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:06:32.101879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"59'9","reported_seq":32,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614355+0000","last_change":"2024-09-10T19:01:42.328462+0000","last_active":"2024-09-10T19:01:52.614355+0000","last_peered":"2024-09-10T19:01:52.614355+0000","last_clean":"2024-09-10T19:01:52.614355+0000","last_became_active":"2024-09-10T19:01:42.327579+0000","last_became_peered":"2024-09-10T19:01:42.327579+0000","last_unstale":"2024-09-10T19:01:52.614355+0000","last_undegraded":"2024-09-10T19:01:52.614355+0000","last_fullsized":"2024-09-10T19:01:52.614355+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:51:59.209933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.608466+0000","last_change":"2024-09-10T19:01:40.317780+0000","last_active":"2024-09-10T19:01:52.608466+0000","last_peered":"2024-09-10T19:01:52.608466+0000","last_clean":"2024-09-10T19:01:52.608466+0000","last_became_active":"2024-09-10T19:01:40.317346+0000","last_became_peered":"2024-09-10T19:01:40.317346+0000","last_unstale":"2024-09-10T19:01:52.608466+0000","last_undegraded":"2024-09-10T19:01:52.608466+0000","last_fullsized":"2024-09-10T19:01:52.608466+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:30:10.923626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976787+0000","last_change":"2024-09-10T19:01:44.347956+0000","last_active":"2024-09-10T19:01:51.976787+0000","last_peered":"2024-09-10T19:01:51.976787+0000","last_clean":"2024-09-10T19:01:51.976787+0000","last_became_active":"2024-09-10T19:01:44.347694+0000","last_became_peered":"2024-09-10T19:01:44.347694+0000","last_unstale":"2024-09-10T19:01:51.976787+0000","last_undegraded":"2024-09-10T19:01:51.976787+0000","last_fullsized":"2024-09-10T19:01:51.976787+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:09:18.783003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335556+0000","last_change":"2024-09-10T19:01:46.347348+0000","last_active":"2024-09-10T19:01:51.335556+0000","last_peered":"2024-09-10T19:01:51.335556+0000","last_clean":"2024-09-10T19:01:51.335556+0000","last_became_active":"2024-09-10T19:01:46.347051+0000","last_became_peered":"2024-09-10T19:01:46.347051+0000","last_unstale":"2024-09-10T19:01:51.335556+0000","last_undegraded":"2024-09-10T19:01:51.335556+0000","last_fullsized":"2024-09-10T19:01:51.335556+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:12:12.230555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"59'11","reported_seq":35,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330313+0000","last_change":"2024-09-10T19:01:42.325262+0000","last_active":"2024-09-10T19:01:51.330313+0000","last_peered":"2024-09-10T19:01:51.330313+0000","last_clean":"2024-09-10T19:01:51.330313+0000","last_became_active":"2024-09-10T19:01:42.324987+0000","last_became_peered":"2024-09-10T19:01:42.324987+0000","last_unstale":"2024-09-10T19:01:51.330313+0000","last_undegraded":"2024-09-10T19:01:51.330313+0000","last_fullsized":"2024-09-10T19:01:51.330313+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:52:03.736308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614862+0000","last_change":"2024-09-10T19:01:40.310304+0000","last_active":"2024-09-10T19:01:52.614862+0000","last_peered":"2024-09-10T19:01:52.614862+0000","last_clean":"2024-09-10T19:01:52.614862+0000","last_became_active":"2024-09-10T19:01:40.310062+0000","last_became_peered":"2024-09-10T19:01:40.310062+0000","last_unstale":"2024-09-10T19:01:52.614862+0000","last_undegraded":"2024-09-10T19:01:52.614862+0000","last_fullsized":"2024-09-10T19:01:52.614862+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:50:16.846872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.613367+0000","last_change":"2024-09-10T19:01:44.347398+0000","last_active":"2024-09-10T19:01:52.613367+0000","last_peered":"2024-09-10T19:01:52.613367+0000","last_clean":"2024-09-10T19:01:52.613367+0000","last_became_active":"2024-09-10T19:01:44.347105+0000","last_became_peered":"2024-09-10T19:01:44.347105+0000","last_unstale":"2024-09-10T19:01:52.613367+0000","last_undegraded":"2024-09-10T19:01:52.613367+0000","last_fullsized":"2024-09-10T19:01:52.613367+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:17:42.033789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976518+0000","last_change":"2024-09-10T19:01:46.343744+0000","last_active":"2024-09-10T19:01:51.976518+0000","last_peered":"2024-09-10T19:01:51.976518+0000","last_clean":"2024-09-10T19:01:51.976518+0000","last_became_active":"2024-09-10T19:01:46.343467+0000","last_became_peered":"2024-09-10T19:01:46.343467+0000","last_unstale":"2024-09-10T19:01:51.976518+0000","last_undegraded":"2024-09-10T19:01:51.976518+0000","last_fullsized":"2024-09-10T19:01:51.976518+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:19:50.625620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"59'4","reported_seq":22,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976679+0000","last_change":"2024-09-10T19:01:42.329213+0000","last_active":"2024-09-10T19:01:51.976679+0000","last_peered":"2024-09-10T19:01:51.976679+0000","last_clean":"2024-09-10T19:01:51.976679+0000","last_became_active":"2024-09-10T19:01:42.328924+0000","last_became_peered":"2024-09-10T19:01:42.328924+0000","last_unstale":"2024-09-10T19:01:51.976679+0000","last_undegraded":"2024-09-10T19:01:51.976679+0000","last_fullsized":"2024-09-10T19:01:51.976679+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:09:00.923719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.347273+0000","last_change":"2024-09-10T19:01:40.316478+0000","last_active":"2024-09-10T19:01:52.347273+0000","last_peered":"2024-09-10T19:01:52.347273+0000","last_clean":"2024-09-10T19:01:52.347273+0000","last_became_active":"2024-09-10T19:01:40.316229+0000","last_became_peered":"2024-09-10T19:01:40.316229+0000","last_unstale":"2024-09-10T19:01:52.347273+0000","last_undegraded":"2024-09-10T19:01:52.347273+0000","last_fullsized":"2024-09-10T19:01:52.347273+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:12:40.874738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.607305+0000","last_change":"2024-09-10T19:01:44.351243+0000","last_active":"2024-09-10T19:01:52.607305+0000","last_peered":"2024-09-10T19:01:52.607305+0000","last_clean":"2024-09-10T19:01:52.607305+0000","last_became_active":"2024-09-10T19:01:44.350871+0000","last_became_peered":"2024-09-10T19:01:44.350871+0000","last_unstale":"2024-09-10T19:01:52.607305+0000","last_undegraded":"2024-09-10T19:01:52.607305+0000","last_fullsized":"2024-09-10T19:01:52.607305+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:35:25.052911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.473901+0000","last_change":"2024-09-10T19:01:46.345647+0000","last_active":"2024-09-10T19:01:52.473901+0000","last_peered":"2024-09-10T19:01:52.473901+0000","last_clean":"2024-09-10T19:01:52.473901+0000","last_became_active":"2024-09-10T19:01:46.345357+0000","last_became_peered":"2024-09-10T19:01:46.345357+0000","last_unstale":"2024-09-10T19:01:52.473901+0000","last_undegraded":"2024-09-10T19:01:52.473901+0000","last_fullsized":"2024-09-10T19:01:52.473901+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:49:09.926505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"59'11","reported_seq":35,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976679+0000","last_change":"2024-09-10T19:01:42.379639+0000","last_active":"2024-09-10T19:01:51.976679+0000","last_peered":"2024-09-10T19:01:51.976679+0000","last_clean":"2024-09-10T19:01:51.976679+0000","last_became_active":"2024-09-10T19:01:42.379006+0000","last_became_peered":"2024-09-10T19:01:42.379006+0000","last_unstale":"2024-09-10T19:01:51.976679+0000","last_undegraded":"2024-09-10T19:01:51.976679+0000","last_fullsized":"2024-09-10T19:01:51.976679+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:34.998663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.609048+0000","last_change":"2024-09-10T19:01:40.317573+0000","last_active":"2024-09-10T19:01:52.609048+0000","last_peered":"2024-09-10T19:01:52.609048+0000","last_clean":"2024-09-10T19:01:52.609048+0000","last_became_active":"2024-09-10T19:01:40.317278+0000","last_became_peered":"2024-09-10T19:01:40.317278+0000","last_unstale":"2024-09-10T19:01:52.609048+0000","last_undegraded":"2024-09-10T19:01:52.609048+0000","last_fullsized":"2024-09-10T19:01:52.609048+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:21:15.544228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.473246+0000","last_change":"2024-09-10T19:01:44.345955+0000","last_active":"2024-09-10T19:01:52.473246+0000","last_peered":"2024-09-10T19:01:52.473246+0000","last_clean":"2024-09-10T19:01:52.473246+0000","last_became_active":"2024-09-10T19:01:44.345674+0000","last_became_peered":"2024-09-10T19:01:44.345674+0000","last_unstale":"2024-09-10T19:01:52.473246+0000","last_undegraded":"2024-09-10T19:01:52.473246+0000","last_fullsized":"2024-09-10T19:01:52.473246+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:34:44.118992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976673+0000","last_change":"2024-09-10T19:01:46.615441+0000","last_active":"2024-09-10T19:01:51.976673+0000","last_peered":"2024-09-10T19:01:51.976673+0000","last_clean":"2024-09-10T19:01:51.976673+0000","last_became_active":"2024-09-10T19:01:46.614382+0000","last_became_peered":"2024-09-10T19:01:46.614382+0000","last_unstale":"2024-09-10T19:01:51.976673+0000","last_undegraded":"2024-09-10T19:01:51.976673+0000","last_fullsized":"2024-09-10T19:01:51.976673+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:28.095017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"59'1","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474200+0000","last_change":"2024-09-10T19:01:46.347709+0000","last_active":"2024-09-10T19:01:52.474200+0000","last_peered":"2024-09-10T19:01:52.474200+0000","last_clean":"2024-09-10T19:01:52.474200+0000","last_became_active":"2024-09-10T19:01:46.347472+0000","last_became_peered":"2024-09-10T19:01:46.347472+0000","last_unstale":"2024-09-10T19:01:52.474200+0000","last_undegraded":"2024-09-10T19:01:52.474200+0000","last_fullsized":"2024-09-10T19:01:52.474200+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:07:55.759839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346625+0000","last_change":"2024-09-10T19:01:40.319612+0000","last_active":"2024-09-10T19:01:52.346625+0000","last_peered":"2024-09-10T19:01:52.346625+0000","last_clean":"2024-09-10T19:01:52.346625+0000","last_became_active":"2024-09-10T19:01:40.319346+0000","last_became_peered":"2024-09-10T19:01:40.319346+0000","last_unstale":"2024-09-10T19:01:52.346625+0000","last_undegraded":"2024-09-10T19:01:52.346625+0000","last_fullsized":"2024-09-10T19:01:52.346625+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:14:42.443756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"59'10","reported_seq":31,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336543+0000","last_change":"2024-09-10T19:01:42.380821+0000","last_active":"2024-09-10T19:01:51.336543+0000","last_peered":"2024-09-10T19:01:51.336543+0000","last_clean":"2024-09-10T19:01:51.336543+0000","last_became_active":"2024-09-10T19:01:42.380475+0000","last_became_peered":"2024-09-10T19:01:42.380475+0000","last_unstale":"2024-09-10T19:01:51.336543+0000","last_undegraded":"2024-09-10T19:01:51.336543+0000","last_fullsized":"2024-09-10T19:01:51.336543+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:00:36.362315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"59'8","reported_seq":41,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:53.778678+0000","last_change":"2024-09-10T19:01:44.352182+0000","last_active":"2024-09-10T19:02:53.778678+0000","last_peered":"2024-09-10T19:02:53.778678+0000","last_clean":"2024-09-10T19:02:53.778678+0000","last_became_active":"2024-09-10T19:01:44.351353+0000","last_became_peered":"2024-09-10T19:01:44.351353+0000","last_unstale":"2024-09-10T19:02:53.778678+0000","last_undegraded":"2024-09-10T19:02:53.778678+0000","last_fullsized":"2024-09-10T19:02:53.778678+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:27.576782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614556+0000","last_change":"2024-09-10T19:01:46.348007+0000","last_active":"2024-09-10T19:01:52.614556+0000","last_peered":"2024-09-10T19:01:52.614556+0000","last_clean":"2024-09-10T19:01:52.614556+0000","last_became_active":"2024-09-10T19:01:46.347710+0000","last_became_peered":"2024-09-10T19:01:46.347710+0000","last_unstale":"2024-09-10T19:01:52.614556+0000","last_undegraded":"2024-09-10T19:01:52.614556+0000","last_fullsized":"2024-09-10T19:01:52.614556+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:51:16.186103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977001+0000","last_change":"2024-09-10T19:01:40.314466+0000","last_active":"2024-09-10T19:01:51.977001+0000","last_peered":"2024-09-10T19:01:51.977001+0000","last_clean":"2024-09-10T19:01:51.977001+0000","last_became_active":"2024-09-10T19:01:40.314031+0000","last_became_peered":"2024-09-10T19:01:40.314031+0000","last_unstale":"2024-09-10T19:01:51.977001+0000","last_undegraded":"2024-09-10T19:01:51.977001+0000","last_fullsized":"2024-09-10T19:01:51.977001+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:08:42.489979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"59'11","reported_seq":35,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.608918+0000","last_change":"2024-09-10T19:01:42.327954+0000","last_active":"2024-09-10T19:01:52.608918+0000","last_peered":"2024-09-10T19:01:52.608918+0000","last_clean":"2024-09-10T19:01:52.608918+0000","last_became_active":"2024-09-10T19:01:42.327652+0000","last_became_peered":"2024-09-10T19:01:42.327652+0000","last_unstale":"2024-09-10T19:01:52.608918+0000","last_undegraded":"2024-09-10T19:01:52.608918+0000","last_fullsized":"2024-09-10T19:01:52.608918+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:13:34.608994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336458+0000","last_change":"2024-09-10T19:01:44.336159+0000","last_active":"2024-09-10T19:01:51.336458+0000","last_peered":"2024-09-10T19:01:51.336458+0000","last_clean":"2024-09-10T19:01:51.336458+0000","last_became_active":"2024-09-10T19:01:44.335815+0000","last_became_peered":"2024-09-10T19:01:44.335815+0000","last_unstale":"2024-09-10T19:01:51.336458+0000","last_undegraded":"2024-09-10T19:01:51.336458+0000","last_fullsized":"2024-09-10T19:01:51.336458+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:43:13.504025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330553+0000","last_change":"2024-09-10T19:01:46.616324+0000","last_active":"2024-09-10T19:01:51.330553+0000","last_peered":"2024-09-10T19:01:51.330553+0000","last_clean":"2024-09-10T19:01:51.330553+0000","last_became_active":"2024-09-10T19:01:46.616045+0000","last_became_peered":"2024-09-10T19:01:46.616045+0000","last_unstale":"2024-09-10T19:01:51.330553+0000","last_undegraded":"2024-09-10T19:01:51.330553+0000","last_fullsized":"2024-09-10T19:01:51.330553+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:23:10.008384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.607768+0000","last_change":"2024-09-10T19:01:40.313651+0000","last_active":"2024-09-10T19:01:52.607768+0000","last_peered":"2024-09-10T19:01:52.607768+0000","last_clean":"2024-09-10T19:01:52.607768+0000","last_became_active":"2024-09-10T19:01:40.313254+0000","last_became_peered":"2024-09-10T19:01:40.313254+0000","last_unstale":"2024-09-10T19:01:52.607768+0000","last_undegraded":"2024-09-10T19:01:52.607768+0000","last_fullsized":"2024-09-10T19:01:52.607768+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:56:10.551627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"59'15","reported_seq":41,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335050+0000","last_change":"2024-09-10T19:01:42.329503+0000","last_active":"2024-09-10T19:01:51.335050+0000","last_peered":"2024-09-10T19:01:51.335050+0000","last_clean":"2024-09-10T19:01:51.335050+0000","last_became_active":"2024-09-10T19:01:42.329219+0000","last_became_peered":"2024-09-10T19:01:42.329219+0000","last_unstale":"2024-09-10T19:01:51.335050+0000","last_undegraded":"2024-09-10T19:01:51.335050+0000","last_fullsized":"2024-09-10T19:01:51.335050+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:14:14.960538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.613506+0000","last_change":"2024-09-10T19:01:44.345686+0000","last_active":"2024-09-10T19:01:52.613506+0000","last_peered":"2024-09-10T19:01:52.613506+0000","last_clean":"2024-09-10T19:01:52.613506+0000","last_became_active":"2024-09-10T19:01:44.345152+0000","last_became_peered":"2024-09-10T19:01:44.345152+0000","last_unstale":"2024-09-10T19:01:52.613506+0000","last_undegraded":"2024-09-10T19:01:52.613506+0000","last_fullsized":"2024-09-10T19:01:52.613506+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:11:53.443027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977453+0000","last_change":"2024-09-10T19:01:46.614616+0000","last_active":"2024-09-10T19:01:51.977453+0000","last_peered":"2024-09-10T19:01:51.977453+0000","last_clean":"2024-09-10T19:01:51.977453+0000","last_became_active":"2024-09-10T19:01:46.614062+0000","last_became_peered":"2024-09-10T19:01:46.614062+0000","last_unstale":"2024-09-10T19:01:51.977453+0000","last_undegraded":"2024-09-10T19:01:51.977453+0000","last_fullsized":"2024-09-10T19:01:51.977453+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:18:46.911945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976995+0000","last_change":"2024-09-10T19:01:40.312236+0000","last_active":"2024-09-10T19:01:51.976995+0000","last_peered":"2024-09-10T19:01:51.976995+0000","last_clean":"2024-09-10T19:01:51.976995+0000","last_became_active":"2024-09-10T19:01:40.311960+0000","last_became_peered":"2024-09-10T19:01:40.311960+0000","last_unstale":"2024-09-10T19:01:51.976995+0000","last_undegraded":"2024-09-10T19:01:51.976995+0000","last_fullsized":"2024-09-10T19:01:51.976995+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:22:12.438528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"59'12","reported_seq":39,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977287+0000","last_change":"2024-09-10T19:01:42.328241+0000","last_active":"2024-09-10T19:01:51.977287+0000","last_peered":"2024-09-10T19:01:51.977287+0000","last_clean":"2024-09-10T19:01:51.977287+0000","last_became_active":"2024-09-10T19:01:42.327964+0000","last_became_peered":"2024-09-10T19:01:42.327964+0000","last_unstale":"2024-09-10T19:01:51.977287+0000","last_undegraded":"2024-09-10T19:01:51.977287+0000","last_fullsized":"2024-09-10T19:01:51.977287+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:58:37.081806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330408+0000","last_change":"2024-09-10T19:01:44.349855+0000","last_active":"2024-09-10T19:01:51.330408+0000","last_peered":"2024-09-10T19:01:51.330408+0000","last_clean":"2024-09-10T19:01:51.330408+0000","last_became_active":"2024-09-10T19:01:44.349548+0000","last_became_peered":"2024-09-10T19:01:44.349548+0000","last_unstale":"2024-09-10T19:01:51.330408+0000","last_undegraded":"2024-09-10T19:01:51.330408+0000","last_fullsized":"2024-09-10T19:01:51.330408+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:06:43.238606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":391,"ondisk_log_size":391,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430230,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298532,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446620,"statfs":{"total":95995035648,"available":95689338880,"internally_reserved":0,"allocated":1208320,"data_stored":822643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.316}]},{"osd":1,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.434}]},{"osd":2,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.443}]},{"osd":3,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.46100000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.371},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.34300000000000003}]},{"osd":5,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.35699999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.29599999999999999}]}]},{"osd":6,"up_from":42,"seq":180388626460,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298512,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446640,"statfs":{"total":95995035648,"available":95689359360,"internally_reserved":0,"allocated":1187840,"data_stored":822042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.57199999999999995}]},{"osd":1,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.51300000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.53100000000000003}]},{"osd":3,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.50700000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.496},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.39600000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.436}]},{"osd":7,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.41199999999999998}]}]},{"osd":5,"up_from":36,"seq":154618822687,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297940,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447212,"statfs":{"total":95995035648,"available":95689945088,"internally_reserved":0,"allocated":598016,"data_stored":231972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48799999999999999,"15min":0.48699999999999999},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.59699999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.498},"min":{"1min":0.374,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.45700000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.42799999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.42899999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.61599999999999999,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.40899999999999997}]},{"osd":3,"last update":"Tue Sep 10 19:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.48999999999999999,"15min":0.48799999999999999},"min":{"1min":0.36499999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63500000000000001,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.438}]},{"osd":4,"last update":"Tue Sep 10 19:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.36199999999999999,"15min":0.35599999999999998},"min":{"1min":0.27700000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.53000000000000003,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.377,"15min":0.36799999999999999},"min":{"1min":0.29199999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.30299999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.32300000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.377},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.315}]}]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297952,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447200,"statfs":{"total":95995035648,"available":95689932800,"internally_reserved":0,"allocated":610304,"data_stored":231798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47499999999999998,"15min":0.47299999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.51900000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46700000000000003,"15min":0.46400000000000002},"min":{"1min":0.35999999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.39800000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71199999999999997,"5min":0.754,"15min":0.754},"last":0.49299999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71199999999999997,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.499,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65300000000000002,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.438}]},{"osd":3,"last update":"Tue Sep 10 19:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.502,"15min":0.501},"min":{"1min":0.35999999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.70299999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.48799999999999999},"min":{"1min":0.371,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.503}]},{"osd":5,"last update":"Tue Sep 10 19:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.54000000000000004,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.46400000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.36299999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.41999999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182441,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297980,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447172,"statfs":{"total":95995035648,"available":95689904128,"internally_reserved":0,"allocated":638976,"data_stored":232839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.36699999999999999,"15min":0.374},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.503,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39600000000000002,"15min":0.40799999999999997},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":0.5,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.39300000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.40100000000000002,"15min":0.42099999999999999},"min":{"1min":0.20000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.48899999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.41099999999999998,"15min":0.42099999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.56100000000000005,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.36499999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42299999999999999,"15min":0.437},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60199999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.39500000000000002,"15min":0.40699999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.497,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.38200000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.66100000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.60399999999999998,"15min":0.61199999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.71099999999999997,"5min":0.878,"15min":0.878},"last":0.69299999999999995}]},{"osd":5,"last update":"Tue Sep 10 19:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.52700000000000002,"15min":0.53100000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.626}]},{"osd":6,"last update":"Tue Sep 10 19:02:02 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.59899999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:02:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.70199999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297940,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447212,"statfs":{"total":95995035648,"available":95689945088,"internally_reserved":0,"allocated":598016,"data_stored":232977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:34 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34699999999999998,"15min":0.35199999999999998},"min":{"1min":0.20200000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.56399999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.376},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.37,"15min":0.36099999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.44900000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:02:34 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.38100000000000001,"15min":0.38300000000000001},"min":{"1min":0.25800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54200000000000004,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.40699999999999997,"15min":0.40899999999999997},"min":{"1min":0.28799999999999998,"5min":0.216,"15min":0.216},"max":{"1min":0.53300000000000003,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.39900000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.442,"15min":0.46000000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61399999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.39600000000000002,"15min":0.41099999999999998},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.48099999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35599999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:02:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.50600000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:02:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.72499999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.497,"15min":0.495},"min":{"1min":0.34799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65700000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.438}]},{"osd":6,"last update":"Tue Sep 10 19:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68899999999999995,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52200000000000002,"15min":0.52300000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69699999999999995,"5min":0.746,"15min":0.746},"last":0.41399999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.46400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297992,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447160,"statfs":{"total":95995035648,"available":95689891840,"internally_reserved":0,"allocated":651264,"data_stored":234148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:09 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34300000000000003,"15min":0.34100000000000003},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.49099999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.224},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35099999999999998,"15min":0.34699999999999998},"min":{"1min":0.23200000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.27500000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:02:35 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.371,"15min":0.36499999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.55400000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.29099999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.41099999999999998,"15min":0.42199999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42799999999999999,"15min":0.437},"min":{"1min":0.28000000000000003,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.33000000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:02:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55300000000000005,"15min":0.55000000000000004},"min":{"1min":0.443,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.39300000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:02:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51100000000000001,"15min":0.51200000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54800000000000004,"15min":0.54400000000000004},"min":{"1min":0.45100000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.69899999999999995,"5min":0.751,"15min":0.751},"last":0.54300000000000004}]},{"osd":6,"last update":"Tue Sep 10 19:02:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.38300000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:02:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.40100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298540,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446612,"statfs":{"total":95995035648,"available":95689330688,"internally_reserved":0,"allocated":1216512,"data_stored":823576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:02:10 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.33200000000000002,"15min":0.312},"min":{"1min":0.23699999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.499,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.371,"5min":0.33700000000000002,"15min":0.317},"min":{"1min":0.245,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.52200000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.28100000000000003}]},{"osd":2,"last update":"Tue Sep 10 19:02:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.38900000000000001,"15min":0.371},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.311},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.38300000000000001,"15min":0.36699999999999999},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.23899999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:02:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.29999999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.376},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44,"15min":0.44},"min":{"1min":0.31900000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.33900000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:02:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54300000000000004,"15min":0.53800000000000003},"min":{"1min":0.374,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.57199999999999995,"15min":0.56599999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.437}]},{"osd":5,"last update":"Tue Sep 10 19:02:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.61099999999999999,"15min":0.61599999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.76600000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.74399999999999999,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.42899999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.44400000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:02:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.38500000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:03:09.339 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:03:09.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:09 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:09.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:09 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:09.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:09 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:09.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:09 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:09.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:09 smithi194 ceph-mon[32464]: Deploying daemon node-exporter.b on smithi194 2024-09-10T19:03:09.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:09 smithi194 ceph-mon[32464]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:03:09.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:09 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:09.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:09 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:09.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:09 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:09.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:09 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:09.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:09 smithi119 ceph-mon[29577]: Deploying daemon node-exporter.b on smithi194 2024-09-10T19:03:09.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:09 smithi119 ceph-mon[29577]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:03:09.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:09 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:09.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:09 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:09.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:09 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:09.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:09 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:09.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:09 smithi119 ceph-mon[40674]: Deploying daemon node-exporter.b on smithi194 2024-09-10T19:03:09.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:09 smithi119 ceph-mon[40674]: from='client.24596 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:03:09.586 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:10.018 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:09 smithi194 bash[66065]: Getting image source signatures 2024-09-10T19:03:10.018 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:09 smithi194 bash[66065]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-10T19:03:10.018 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:09 smithi194 bash[66065]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-10T19:03:10.019 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:09 smithi194 bash[66065]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-10T19:03:10.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:10 smithi194 ceph-mon[32464]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:10.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:03:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:03:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:03:10.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:10 smithi119 ceph-mon[29577]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:10.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:10 smithi119 ceph-mon[40674]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:11.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.532+0000 7f6048bce640 1 -- 172.21.15.119:0/3046981060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60440fb9c0 msgr2=0x7f60440fbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.532+0000 7f6048bce640 1 --2- 172.21.15.119:0/3046981060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60440fb9c0 0x7f60440fbe10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6038004890 tx=0x7f6038009da0 comp rx=0 tx=0).stop 2024-09-10T19:03:11.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 -- 172.21.15.119:0/3046981060 shutdown_connections 2024-09-10T19:03:11.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 --2- 172.21.15.119:0/3046981060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60440fc350 0x7f60441085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:11.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 --2- 172.21.15.119:0/3046981060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60440fb9c0 0x7f60440fbe10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:11.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 --2- 172.21.15.119:0/3046981060 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6044101850 0x7f6044101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:11.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 -- 172.21.15.119:0/3046981060 >> 172.21.15.119:0/3046981060 conn(0x7f60440724b0 msgr2=0x7f60440728b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:11.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 -- 172.21.15.119:0/3046981060 shutdown_connections 2024-09-10T19:03:11.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 -- 172.21.15.119:0/3046981060 wait complete. 2024-09-10T19:03:11.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 Processor -- start 2024-09-10T19:03:11.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 -- start start 2024-09-10T19:03:11.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f60440fb9c0 0x7f6044112da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:11.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60440fc350 0x7f60441132e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:11.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6044101850 0x7f604410cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:11.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604410d620 con 0x7f6044101850 2024-09-10T19:03:11.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f604410d790 con 0x7f60440fc350 2024-09-10T19:03:11.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.533+0000 7f6048bce640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f604410d950 con 0x7f60440fb9c0 2024-09-10T19:03:11.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f604259c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f60440fb9c0 0x7f6044112da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:11.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6042d9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6044101850 0x7f604410cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:11.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f604259c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f60440fb9c0 0x7f6044112da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:60874/0 (socket says 172.21.15.119:60874) 2024-09-10T19:03:11.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6042d9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6044101850 0x7f604410cfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40574/0 (socket says 172.21.15.119:40574) 2024-09-10T19:03:11.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6042d9d640 1 -- 172.21.15.119:0/1065581402 learned_addr learned my addr 172.21.15.119:0/1065581402 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:03:11.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6041d9b640 1 --2- 172.21.15.119:0/1065581402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60440fc350 0x7f60441132e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:11.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6042d9d640 1 -- 172.21.15.119:0/1065581402 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f60440fb9c0 msgr2=0x7f6044112da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6042d9d640 1 --2- 172.21.15.119:0/1065581402 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f60440fb9c0 0x7f6044112da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:11.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6042d9d640 1 -- 172.21.15.119:0/1065581402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60440fc350 msgr2=0x7f60441132e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6042d9d640 1 --2- 172.21.15.119:0/1065581402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60440fc350 0x7f60441132e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:11.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6042d9d640 1 -- 172.21.15.119:0/1065581402 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6034005040 con 0x7f6044101850 2024-09-10T19:03:11.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6042d9d640 1 --2- 172.21.15.119:0/1065581402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6044101850 0x7f604410cfd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f603c00ece0 tx=0x7f603c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:03:11.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6032ffd640 1 -- 172.21.15.119:0/1065581402 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c00ee40 con 0x7f6044101850 2024-09-10T19:03:11.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6032ffd640 1 -- 172.21.15.119:0/1065581402 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f603c00f040 con 0x7f6044101850 2024-09-10T19:03:11.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6048bce640 1 -- 172.21.15.119:0/1065581402 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6038005040 con 0x7f6044101850 2024-09-10T19:03:11.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6032ffd640 1 -- 172.21.15.119:0/1065581402 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f603c009420 con 0x7f6044101850 2024-09-10T19:03:11.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.534+0000 7f6048bce640 1 -- 172.21.15.119:0/1065581402 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6044119bf0 con 0x7f6044101850 2024-09-10T19:03:11.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.535+0000 7f6048bce640 1 -- 172.21.15.119:0/1065581402 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60440fcf70 con 0x7f6044101850 2024-09-10T19:03:11.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.536+0000 7f6032ffd640 1 -- 172.21.15.119:0/1065581402 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f603c007890 con 0x7f6044101850 2024-09-10T19:03:11.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.536+0000 7f6032ffd640 1 --2- 172.21.15.119:0/1065581402 >> v2:172.21.15.119:6800/3118961733 conn(0x7f602c06eed0 0x7f602c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:11.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.536+0000 7f6032ffd640 1 -- 172.21.15.119:0/1065581402 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f603c08eae0 con 0x7f6044101850 2024-09-10T19:03:11.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.536+0000 7f604259c640 1 --2- 172.21.15.119:0/1065581402 >> v2:172.21.15.119:6800/3118961733 conn(0x7f602c06eed0 0x7f602c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:11.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.537+0000 7f604259c640 1 --2- 172.21.15.119:0/1065581402 >> v2:172.21.15.119:6800/3118961733 conn(0x7f602c06eed0 0x7f602c071380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6034007fd0 tx=0x7f603400b040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:03:11.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.540+0000 7f6032ffd640 1 -- 172.21.15.119:0/1065581402 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f603c05b9f0 con 0x7f6044101850 2024-09-10T19:03:11.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.694+0000 7f6048bce640 1 -- 172.21.15.119:0/1065581402 --> v2:172.21.15.119:6800/3118961733 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f604410ac90 con 0x7f602c06eed0 2024-09-10T19:03:11.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.703+0000 7f6032ffd640 1 -- 172.21.15.119:0/1065581402 <== mgr.24452 v2:172.21.15.119:6800/3118961733 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+383694 (secure 0 0 0) 0x7f604410ac90 con 0x7f602c06eed0 2024-09-10T19:03:11.709 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T19:03:11.711 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-10T19:03:11.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.707+0000 7f6048bce640 1 -- 172.21.15.119:0/1065581402 >> v2:172.21.15.119:6800/3118961733 conn(0x7f602c06eed0 msgr2=0x7f602c071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.707+0000 7f6048bce640 1 --2- 172.21.15.119:0/1065581402 >> v2:172.21.15.119:6800/3118961733 conn(0x7f602c06eed0 0x7f602c071380 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6034007fd0 tx=0x7f603400b040 comp rx=0 tx=0).stop 2024-09-10T19:03:11.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.707+0000 7f6048bce640 1 -- 172.21.15.119:0/1065581402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6044101850 msgr2=0x7f604410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:11.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.707+0000 7f6048bce640 1 --2- 172.21.15.119:0/1065581402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6044101850 0x7f604410cfd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f603c00ece0 tx=0x7f603c00b040 comp rx=0 tx=0).stop 2024-09-10T19:03:11.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.707+0000 7f6048bce640 1 -- 172.21.15.119:0/1065581402 shutdown_connections 2024-09-10T19:03:11.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.707+0000 7f6048bce640 1 --2- 172.21.15.119:0/1065581402 >> v2:172.21.15.119:6800/3118961733 conn(0x7f602c06eed0 0x7f602c071380 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:11.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.707+0000 7f6048bce640 1 --2- 172.21.15.119:0/1065581402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6044101850 0x7f604410cfd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:11.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.707+0000 7f6048bce640 1 --2- 172.21.15.119:0/1065581402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60440fc350 0x7f60441132e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:11.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.707+0000 7f6048bce640 1 --2- 172.21.15.119:0/1065581402 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f60440fb9c0 0x7f6044112da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:11.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.707+0000 7f6048bce640 1 -- 172.21.15.119:0/1065581402 >> 172.21.15.119:0/1065581402 conn(0x7f60440724b0 msgr2=0x7f6044065f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:11.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.707+0000 7f6048bce640 1 -- 172.21.15.119:0/1065581402 shutdown_connections 2024-09-10T19:03:11.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:11.707+0000 7f6048bce640 1 -- 172.21.15.119:0/1065581402 wait complete. 2024-09-10T19:03:12.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:12 smithi194 ceph-mon[32464]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:12.550 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:12 smithi119 ceph-mon[29577]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:12.550 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:12 smithi119 ceph-mon[40674]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:12.578 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-09-10T19:03:11.038530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":934,"num_read_kb":791,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":548,"ondisk_log_size":548,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2385388,"kb_used_data":6552,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747575828,"statfs":{"total":767960285184,"available":765517647872,"internally_reserved":0,"allocated":6709248,"data_stored":3631995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003118"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976690+0000","last_change":"2024-09-10T19:01:46.615905+0000","last_active":"2024-09-10T19:01:51.976690+0000","last_peered":"2024-09-10T19:01:51.976690+0000","last_clean":"2024-09-10T19:01:51.976690+0000","last_became_active":"2024-09-10T19:01:46.615569+0000","last_became_peered":"2024-09-10T19:01:46.615569+0000","last_unstale":"2024-09-10T19:01:51.976690+0000","last_undegraded":"2024-09-10T19:01:51.976690+0000","last_fullsized":"2024-09-10T19:01:51.976690+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:40:12.111111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336575+0000","last_change":"2024-09-10T19:01:40.316539+0000","last_active":"2024-09-10T19:01:51.336575+0000","last_peered":"2024-09-10T19:01:51.336575+0000","last_clean":"2024-09-10T19:01:51.336575+0000","last_became_active":"2024-09-10T19:01:40.316258+0000","last_became_peered":"2024-09-10T19:01:40.316258+0000","last_unstale":"2024-09-10T19:01:51.336575+0000","last_undegraded":"2024-09-10T19:01:51.336575+0000","last_fullsized":"2024-09-10T19:01:51.336575+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:11:06.642347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"59'15","reported_seq":41,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976906+0000","last_change":"2024-09-10T19:01:42.325766+0000","last_active":"2024-09-10T19:01:51.976906+0000","last_peered":"2024-09-10T19:01:51.976906+0000","last_clean":"2024-09-10T19:01:51.976906+0000","last_became_active":"2024-09-10T19:01:42.325498+0000","last_became_peered":"2024-09-10T19:01:42.325498+0000","last_unstale":"2024-09-10T19:01:51.976906+0000","last_undegraded":"2024-09-10T19:01:51.976906+0000","last_fullsized":"2024-09-10T19:01:51.976906+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:31:45.451249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330047+0000","last_change":"2024-09-10T19:01:44.345969+0000","last_active":"2024-09-10T19:01:51.330047+0000","last_peered":"2024-09-10T19:01:51.330047+0000","last_clean":"2024-09-10T19:01:51.330047+0000","last_became_active":"2024-09-10T19:01:44.345702+0000","last_became_peered":"2024-09-10T19:01:44.345702+0000","last_unstale":"2024-09-10T19:01:51.330047+0000","last_undegraded":"2024-09-10T19:01:51.330047+0000","last_fullsized":"2024-09-10T19:01:51.330047+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:21.166785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"52'1","reported_seq":25,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977457+0000","last_change":"2024-09-10T19:01:40.315403+0000","last_active":"2024-09-10T19:01:51.977457+0000","last_peered":"2024-09-10T19:01:51.977457+0000","last_clean":"2024-09-10T19:01:51.977457+0000","last_became_active":"2024-09-10T19:01:40.314931+0000","last_became_peered":"2024-09-10T19:01:40.314931+0000","last_unstale":"2024-09-10T19:01:51.977457+0000","last_undegraded":"2024-09-10T19:01:51.977457+0000","last_fullsized":"2024-09-10T19:01:51.977457+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:11.025210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"59'9","reported_seq":32,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330086+0000","last_change":"2024-09-10T19:01:42.324000+0000","last_active":"2024-09-10T19:01:51.330086+0000","last_peered":"2024-09-10T19:01:51.330086+0000","last_clean":"2024-09-10T19:01:51.330086+0000","last_became_active":"2024-09-10T19:01:42.323673+0000","last_became_peered":"2024-09-10T19:01:42.323673+0000","last_unstale":"2024-09-10T19:01:51.330086+0000","last_undegraded":"2024-09-10T19:01:51.330086+0000","last_fullsized":"2024-09-10T19:01:51.330086+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:53:38.000729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.613366+0000","last_change":"2024-09-10T19:01:44.347117+0000","last_active":"2024-09-10T19:01:52.613366+0000","last_peered":"2024-09-10T19:01:52.613366+0000","last_clean":"2024-09-10T19:01:52.613366+0000","last_became_active":"2024-09-10T19:01:44.346786+0000","last_became_peered":"2024-09-10T19:01:44.346786+0000","last_unstale":"2024-09-10T19:01:52.613366+0000","last_undegraded":"2024-09-10T19:01:52.613366+0000","last_fullsized":"2024-09-10T19:01:52.613366+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:47:10.722092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330149+0000","last_change":"2024-09-10T19:01:46.344623+0000","last_active":"2024-09-10T19:01:51.330149+0000","last_peered":"2024-09-10T19:01:51.330149+0000","last_clean":"2024-09-10T19:01:51.330149+0000","last_became_active":"2024-09-10T19:01:46.344322+0000","last_became_peered":"2024-09-10T19:01:46.344322+0000","last_unstale":"2024-09-10T19:01:51.330149+0000","last_undegraded":"2024-09-10T19:01:51.330149+0000","last_fullsized":"2024-09-10T19:01:51.330149+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:37.837564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474676+0000","last_change":"2024-09-10T19:01:40.317519+0000","last_active":"2024-09-10T19:01:52.474676+0000","last_peered":"2024-09-10T19:01:52.474676+0000","last_clean":"2024-09-10T19:01:52.474676+0000","last_became_active":"2024-09-10T19:01:40.317117+0000","last_became_peered":"2024-09-10T19:01:40.317117+0000","last_unstale":"2024-09-10T19:01:52.474676+0000","last_undegraded":"2024-09-10T19:01:52.474676+0000","last_fullsized":"2024-09-10T19:01:52.474676+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:00:25.740792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"59'5","reported_seq":26,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330232+0000","last_change":"2024-09-10T19:01:42.325747+0000","last_active":"2024-09-10T19:01:51.330232+0000","last_peered":"2024-09-10T19:01:51.330232+0000","last_clean":"2024-09-10T19:01:51.330232+0000","last_became_active":"2024-09-10T19:01:42.325386+0000","last_became_peered":"2024-09-10T19:01:42.325386+0000","last_unstale":"2024-09-10T19:01:51.330232+0000","last_undegraded":"2024-09-10T19:01:51.330232+0000","last_fullsized":"2024-09-10T19:01:51.330232+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:53.254045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.473269+0000","last_change":"2024-09-10T19:01:44.340522+0000","last_active":"2024-09-10T19:01:52.473269+0000","last_peered":"2024-09-10T19:01:52.473269+0000","last_clean":"2024-09-10T19:01:52.473269+0000","last_became_active":"2024-09-10T19:01:44.340252+0000","last_became_peered":"2024-09-10T19:01:44.340252+0000","last_unstale":"2024-09-10T19:01:52.473269+0000","last_undegraded":"2024-09-10T19:01:52.473269+0000","last_fullsized":"2024-09-10T19:01:52.473269+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:31:27.553639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346753+0000","last_change":"2024-09-10T19:01:46.346256+0000","last_active":"2024-09-10T19:01:52.346753+0000","last_peered":"2024-09-10T19:01:52.346753+0000","last_clean":"2024-09-10T19:01:52.346753+0000","last_became_active":"2024-09-10T19:01:46.346005+0000","last_became_peered":"2024-09-10T19:01:46.346005+0000","last_unstale":"2024-09-10T19:01:52.346753+0000","last_undegraded":"2024-09-10T19:01:52.346753+0000","last_fullsized":"2024-09-10T19:01:52.346753+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:20.283012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474391+0000","last_change":"2024-09-10T19:01:40.319607+0000","last_active":"2024-09-10T19:01:52.474391+0000","last_peered":"2024-09-10T19:01:52.474391+0000","last_clean":"2024-09-10T19:01:52.474391+0000","last_became_active":"2024-09-10T19:01:40.319363+0000","last_became_peered":"2024-09-10T19:01:40.319363+0000","last_unstale":"2024-09-10T19:01:52.474391+0000","last_undegraded":"2024-09-10T19:01:52.474391+0000","last_fullsized":"2024-09-10T19:01:52.474391+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:33.978961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"59'9","reported_seq":32,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330301+0000","last_change":"2024-09-10T19:01:42.321058+0000","last_active":"2024-09-10T19:01:51.330301+0000","last_peered":"2024-09-10T19:01:51.330301+0000","last_clean":"2024-09-10T19:01:51.330301+0000","last_became_active":"2024-09-10T19:01:42.320753+0000","last_became_peered":"2024-09-10T19:01:42.320753+0000","last_unstale":"2024-09-10T19:01:51.330301+0000","last_undegraded":"2024-09-10T19:01:51.330301+0000","last_fullsized":"2024-09-10T19:01:51.330301+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:42:38.537340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346436+0000","last_change":"2024-09-10T19:01:44.343628+0000","last_active":"2024-09-10T19:01:52.346436+0000","last_peered":"2024-09-10T19:01:52.346436+0000","last_clean":"2024-09-10T19:01:52.346436+0000","last_became_active":"2024-09-10T19:01:44.343359+0000","last_became_peered":"2024-09-10T19:01:44.343359+0000","last_unstale":"2024-09-10T19:01:52.346436+0000","last_undegraded":"2024-09-10T19:01:52.346436+0000","last_fullsized":"2024-09-10T19:01:52.346436+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:23:11.270396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335744+0000","last_change":"2024-09-10T19:01:46.347691+0000","last_active":"2024-09-10T19:01:51.335744+0000","last_peered":"2024-09-10T19:01:51.335744+0000","last_clean":"2024-09-10T19:01:51.335744+0000","last_became_active":"2024-09-10T19:01:46.347366+0000","last_became_peered":"2024-09-10T19:01:46.347366+0000","last_unstale":"2024-09-10T19:01:51.335744+0000","last_undegraded":"2024-09-10T19:01:51.335744+0000","last_fullsized":"2024-09-10T19:01:51.335744+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:01:01.695933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"59'17","reported_seq":44,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330743+0000","last_change":"2024-09-10T19:01:42.325217+0000","last_active":"2024-09-10T19:01:51.330743+0000","last_peered":"2024-09-10T19:01:51.330743+0000","last_clean":"2024-09-10T19:01:51.330743+0000","last_became_active":"2024-09-10T19:01:42.324863+0000","last_became_peered":"2024-09-10T19:01:42.324863+0000","last_unstale":"2024-09-10T19:01:51.330743+0000","last_undegraded":"2024-09-10T19:01:51.330743+0000","last_fullsized":"2024-09-10T19:01:51.330743+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:58:09.129606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474339+0000","last_change":"2024-09-10T19:01:40.320087+0000","last_active":"2024-09-10T19:01:52.474339+0000","last_peered":"2024-09-10T19:01:52.474339+0000","last_clean":"2024-09-10T19:01:52.474339+0000","last_became_active":"2024-09-10T19:01:40.319542+0000","last_became_peered":"2024-09-10T19:01:40.319542+0000","last_unstale":"2024-09-10T19:01:52.474339+0000","last_undegraded":"2024-09-10T19:01:52.474339+0000","last_fullsized":"2024-09-10T19:01:52.474339+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:59.316079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.613469+0000","last_change":"2024-09-10T19:01:44.345228+0000","last_active":"2024-09-10T19:01:52.613469+0000","last_peered":"2024-09-10T19:01:52.613469+0000","last_clean":"2024-09-10T19:01:52.613469+0000","last_became_active":"2024-09-10T19:01:44.344810+0000","last_became_peered":"2024-09-10T19:01:44.344810+0000","last_unstale":"2024-09-10T19:01:52.613469+0000","last_undegraded":"2024-09-10T19:01:52.613469+0000","last_fullsized":"2024-09-10T19:01:52.613469+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:54:04.769483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335157+0000","last_change":"2024-09-10T19:01:46.343343+0000","last_active":"2024-09-10T19:01:51.335157+0000","last_peered":"2024-09-10T19:01:51.335157+0000","last_clean":"2024-09-10T19:01:51.335157+0000","last_became_active":"2024-09-10T19:01:46.343035+0000","last_became_peered":"2024-09-10T19:01:46.343035+0000","last_unstale":"2024-09-10T19:01:51.335157+0000","last_undegraded":"2024-09-10T19:01:51.335157+0000","last_fullsized":"2024-09-10T19:01:51.335157+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:58:45.098178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"59'10","reported_seq":31,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330458+0000","last_change":"2024-09-10T19:01:42.322765+0000","last_active":"2024-09-10T19:01:51.330458+0000","last_peered":"2024-09-10T19:01:51.330458+0000","last_clean":"2024-09-10T19:01:51.330458+0000","last_became_active":"2024-09-10T19:01:42.322436+0000","last_became_peered":"2024-09-10T19:01:42.322436+0000","last_unstale":"2024-09-10T19:01:51.330458+0000","last_undegraded":"2024-09-10T19:01:51.330458+0000","last_fullsized":"2024-09-10T19:01:51.330458+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:09:55.001758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614549+0000","last_change":"2024-09-10T19:01:40.316538+0000","last_active":"2024-09-10T19:01:52.614549+0000","last_peered":"2024-09-10T19:01:52.614549+0000","last_clean":"2024-09-10T19:01:52.614549+0000","last_became_active":"2024-09-10T19:01:40.315880+0000","last_became_peered":"2024-09-10T19:01:40.315880+0000","last_unstale":"2024-09-10T19:01:52.614549+0000","last_undegraded":"2024-09-10T19:01:52.614549+0000","last_fullsized":"2024-09-10T19:01:52.614549+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:03:06.440224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"59'8","reported_seq":39,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:58.778808+0000","last_change":"2024-09-10T19:01:44.340940+0000","last_active":"2024-09-10T19:02:58.778808+0000","last_peered":"2024-09-10T19:02:58.778808+0000","last_clean":"2024-09-10T19:02:58.778808+0000","last_became_active":"2024-09-10T19:01:44.339818+0000","last_became_peered":"2024-09-10T19:01:44.339818+0000","last_unstale":"2024-09-10T19:02:58.778808+0000","last_undegraded":"2024-09-10T19:02:58.778808+0000","last_fullsized":"2024-09-10T19:02:58.778808+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:15:31.816876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330517+0000","last_change":"2024-09-10T19:01:46.346880+0000","last_active":"2024-09-10T19:01:51.330517+0000","last_peered":"2024-09-10T19:01:51.330517+0000","last_clean":"2024-09-10T19:01:51.330517+0000","last_became_active":"2024-09-10T19:01:46.346609+0000","last_became_peered":"2024-09-10T19:01:46.346609+0000","last_unstale":"2024-09-10T19:01:51.330517+0000","last_undegraded":"2024-09-10T19:01:51.330517+0000","last_fullsized":"2024-09-10T19:01:51.330517+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:28:14.026153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"59'15","reported_seq":41,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.615071+0000","last_change":"2024-09-10T19:01:42.330612+0000","last_active":"2024-09-10T19:01:52.615071+0000","last_peered":"2024-09-10T19:01:52.615071+0000","last_clean":"2024-09-10T19:01:52.615071+0000","last_became_active":"2024-09-10T19:01:42.330264+0000","last_became_peered":"2024-09-10T19:01:42.330264+0000","last_unstale":"2024-09-10T19:01:52.615071+0000","last_undegraded":"2024-09-10T19:01:52.615071+0000","last_fullsized":"2024-09-10T19:01:52.615071+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:22:26.401805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.613902+0000","last_change":"2024-09-10T19:01:40.315446+0000","last_active":"2024-09-10T19:01:52.613902+0000","last_peered":"2024-09-10T19:01:52.613902+0000","last_clean":"2024-09-10T19:01:52.613902+0000","last_became_active":"2024-09-10T19:01:40.315185+0000","last_became_peered":"2024-09-10T19:01:40.315185+0000","last_unstale":"2024-09-10T19:01:52.613902+0000","last_undegraded":"2024-09-10T19:01:52.613902+0000","last_fullsized":"2024-09-10T19:01:52.613902+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:12:52.822736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"59'8","reported_seq":39,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:58.778976+0000","last_change":"2024-09-10T19:01:44.350077+0000","last_active":"2024-09-10T19:02:58.778976+0000","last_peered":"2024-09-10T19:02:58.778976+0000","last_clean":"2024-09-10T19:02:58.778976+0000","last_became_active":"2024-09-10T19:01:44.349746+0000","last_became_peered":"2024-09-10T19:01:44.349746+0000","last_unstale":"2024-09-10T19:02:58.778976+0000","last_undegraded":"2024-09-10T19:02:58.778976+0000","last_fullsized":"2024-09-10T19:02:58.778976+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:59.302630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.347086+0000","last_change":"2024-09-10T19:01:46.347513+0000","last_active":"2024-09-10T19:01:52.347086+0000","last_peered":"2024-09-10T19:01:52.347086+0000","last_clean":"2024-09-10T19:01:52.347086+0000","last_became_active":"2024-09-10T19:01:46.347277+0000","last_became_peered":"2024-09-10T19:01:46.347277+0000","last_unstale":"2024-09-10T19:01:52.347086+0000","last_undegraded":"2024-09-10T19:01:52.347086+0000","last_fullsized":"2024-09-10T19:01:52.347086+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:32:53.136513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"59'11","reported_seq":35,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330627+0000","last_change":"2024-09-10T19:01:42.325306+0000","last_active":"2024-09-10T19:01:51.330627+0000","last_peered":"2024-09-10T19:01:51.330627+0000","last_clean":"2024-09-10T19:01:51.330627+0000","last_became_active":"2024-09-10T19:01:42.325044+0000","last_became_peered":"2024-09-10T19:01:42.325044+0000","last_unstale":"2024-09-10T19:01:51.330627+0000","last_undegraded":"2024-09-10T19:01:51.330627+0000","last_fullsized":"2024-09-10T19:01:51.330627+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:34:05.813571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.473878+0000","last_change":"2024-09-10T19:01:40.322115+0000","last_active":"2024-09-10T19:01:52.473878+0000","last_peered":"2024-09-10T19:01:52.473878+0000","last_clean":"2024-09-10T19:01:52.473878+0000","last_became_active":"2024-09-10T19:01:40.321510+0000","last_became_peered":"2024-09-10T19:01:40.321510+0000","last_unstale":"2024-09-10T19:01:52.473878+0000","last_undegraded":"2024-09-10T19:01:52.473878+0000","last_fullsized":"2024-09-10T19:01:52.473878+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:22:12.047819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346704+0000","last_change":"2024-09-10T19:01:44.344680+0000","last_active":"2024-09-10T19:01:52.346704+0000","last_peered":"2024-09-10T19:01:52.346704+0000","last_clean":"2024-09-10T19:01:52.346704+0000","last_became_active":"2024-09-10T19:01:44.344390+0000","last_became_peered":"2024-09-10T19:01:44.344390+0000","last_unstale":"2024-09-10T19:01:52.346704+0000","last_undegraded":"2024-09-10T19:01:52.346704+0000","last_fullsized":"2024-09-10T19:01:52.346704+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:25:17.312301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976932+0000","last_change":"2024-09-10T19:01:46.614346+0000","last_active":"2024-09-10T19:01:51.976932+0000","last_peered":"2024-09-10T19:01:51.976932+0000","last_clean":"2024-09-10T19:01:51.976932+0000","last_became_active":"2024-09-10T19:01:46.613951+0000","last_became_peered":"2024-09-10T19:01:46.613951+0000","last_unstale":"2024-09-10T19:01:51.976932+0000","last_undegraded":"2024-09-10T19:01:51.976932+0000","last_fullsized":"2024-09-10T19:01:51.976932+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:32:16.513970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"59'14","reported_seq":37,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330199+0000","last_change":"2024-09-10T19:01:42.326162+0000","last_active":"2024-09-10T19:01:51.330199+0000","last_peered":"2024-09-10T19:01:51.330199+0000","last_clean":"2024-09-10T19:01:51.330199+0000","last_became_active":"2024-09-10T19:01:42.325837+0000","last_became_peered":"2024-09-10T19:01:42.325837+0000","last_unstale":"2024-09-10T19:01:51.330199+0000","last_undegraded":"2024-09-10T19:01:51.330199+0000","last_fullsized":"2024-09-10T19:01:51.330199+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:41:15.681351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.609066+0000","last_change":"2024-09-10T19:01:40.313839+0000","last_active":"2024-09-10T19:01:52.609066+0000","last_peered":"2024-09-10T19:01:52.609066+0000","last_clean":"2024-09-10T19:01:52.609066+0000","last_became_active":"2024-09-10T19:01:40.313296+0000","last_became_peered":"2024-09-10T19:01:40.313296+0000","last_unstale":"2024-09-10T19:01:52.609066+0000","last_undegraded":"2024-09-10T19:01:52.609066+0000","last_fullsized":"2024-09-10T19:01:52.609066+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:44:00.974249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"59'8","reported_seq":39,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:58.778956+0000","last_change":"2024-09-10T19:01:44.349408+0000","last_active":"2024-09-10T19:02:58.778956+0000","last_peered":"2024-09-10T19:02:58.778956+0000","last_clean":"2024-09-10T19:02:58.778956+0000","last_became_active":"2024-09-10T19:01:44.349080+0000","last_became_peered":"2024-09-10T19:01:44.349080+0000","last_unstale":"2024-09-10T19:02:58.778956+0000","last_undegraded":"2024-09-10T19:02:58.778956+0000","last_fullsized":"2024-09-10T19:02:58.778956+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:56:31.359086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474222+0000","last_change":"2024-09-10T19:01:46.614432+0000","last_active":"2024-09-10T19:01:52.474222+0000","last_peered":"2024-09-10T19:01:52.474222+0000","last_clean":"2024-09-10T19:01:52.474222+0000","last_became_active":"2024-09-10T19:01:46.614179+0000","last_became_peered":"2024-09-10T19:01:46.614179+0000","last_unstale":"2024-09-10T19:01:52.474222+0000","last_undegraded":"2024-09-10T19:01:52.474222+0000","last_fullsized":"2024-09-10T19:01:52.474222+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:43:41.707412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"59'18","reported_seq":48,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976646+0000","last_change":"2024-09-10T19:01:42.381843+0000","last_active":"2024-09-10T19:01:51.976646+0000","last_peered":"2024-09-10T19:01:51.976646+0000","last_clean":"2024-09-10T19:01:51.976646+0000","last_became_active":"2024-09-10T19:01:42.381564+0000","last_became_peered":"2024-09-10T19:01:42.381564+0000","last_unstale":"2024-09-10T19:01:51.976646+0000","last_undegraded":"2024-09-10T19:01:51.976646+0000","last_fullsized":"2024-09-10T19:01:51.976646+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:53:40.137432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614351+0000","last_change":"2024-09-10T19:01:40.310806+0000","last_active":"2024-09-10T19:01:52.614351+0000","last_peered":"2024-09-10T19:01:52.614351+0000","last_clean":"2024-09-10T19:01:52.614351+0000","last_became_active":"2024-09-10T19:01:40.310526+0000","last_became_peered":"2024-09-10T19:01:40.310526+0000","last_unstale":"2024-09-10T19:01:52.614351+0000","last_undegraded":"2024-09-10T19:01:52.614351+0000","last_fullsized":"2024-09-10T19:01:52.614351+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:08:45.808788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330069+0000","last_change":"2024-09-10T19:01:44.344688+0000","last_active":"2024-09-10T19:01:51.330069+0000","last_peered":"2024-09-10T19:01:51.330069+0000","last_clean":"2024-09-10T19:01:51.330069+0000","last_became_active":"2024-09-10T19:01:44.344387+0000","last_became_peered":"2024-09-10T19:01:44.344387+0000","last_unstale":"2024-09-10T19:01:51.330069+0000","last_undegraded":"2024-09-10T19:01:51.330069+0000","last_fullsized":"2024-09-10T19:01:51.330069+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:35:31.388096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330246+0000","last_change":"2024-09-10T19:01:46.346430+0000","last_active":"2024-09-10T19:01:51.330246+0000","last_peered":"2024-09-10T19:01:51.330246+0000","last_clean":"2024-09-10T19:01:51.330246+0000","last_became_active":"2024-09-10T19:01:46.346135+0000","last_became_peered":"2024-09-10T19:01:46.346135+0000","last_unstale":"2024-09-10T19:01:51.330246+0000","last_undegraded":"2024-09-10T19:01:51.330246+0000","last_fullsized":"2024-09-10T19:01:51.330246+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:09:40.356164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"59'19","reported_seq":52,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336397+0000","last_change":"2024-09-10T19:01:42.379107+0000","last_active":"2024-09-10T19:01:51.336397+0000","last_peered":"2024-09-10T19:01:51.336397+0000","last_clean":"2024-09-10T19:01:51.336397+0000","last_became_active":"2024-09-10T19:01:42.378807+0000","last_became_peered":"2024-09-10T19:01:42.378807+0000","last_unstale":"2024-09-10T19:01:51.336397+0000","last_undegraded":"2024-09-10T19:01:51.336397+0000","last_fullsized":"2024-09-10T19:01:51.336397+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:21:08.942965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474690+0000","last_change":"2024-09-10T19:01:40.318315+0000","last_active":"2024-09-10T19:01:52.474690+0000","last_peered":"2024-09-10T19:01:52.474690+0000","last_clean":"2024-09-10T19:01:52.474690+0000","last_became_active":"2024-09-10T19:01:40.318059+0000","last_became_peered":"2024-09-10T19:01:40.318059+0000","last_unstale":"2024-09-10T19:01:52.474690+0000","last_undegraded":"2024-09-10T19:01:52.474690+0000","last_fullsized":"2024-09-10T19:01:52.474690+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:24:29.750728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.607282+0000","last_change":"2024-09-10T19:01:44.345624+0000","last_active":"2024-09-10T19:01:52.607282+0000","last_peered":"2024-09-10T19:01:52.607282+0000","last_clean":"2024-09-10T19:01:52.607282+0000","last_became_active":"2024-09-10T19:01:44.345282+0000","last_became_peered":"2024-09-10T19:01:44.345282+0000","last_unstale":"2024-09-10T19:01:52.607282+0000","last_undegraded":"2024-09-10T19:01:52.607282+0000","last_fullsized":"2024-09-10T19:01:52.607282+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:45.825160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.613698+0000","last_change":"2024-09-10T19:01:46.614413+0000","last_active":"2024-09-10T19:01:52.613698+0000","last_peered":"2024-09-10T19:01:52.613698+0000","last_clean":"2024-09-10T19:01:52.613698+0000","last_became_active":"2024-09-10T19:01:46.613982+0000","last_became_peered":"2024-09-10T19:01:46.613982+0000","last_unstale":"2024-09-10T19:01:52.613698+0000","last_undegraded":"2024-09-10T19:01:52.613698+0000","last_fullsized":"2024-09-10T19:01:52.613698+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:16:42.295482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"59'10","reported_seq":31,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614429+0000","last_change":"2024-09-10T19:01:42.326777+0000","last_active":"2024-09-10T19:01:52.614429+0000","last_peered":"2024-09-10T19:01:52.614429+0000","last_clean":"2024-09-10T19:01:52.614429+0000","last_became_active":"2024-09-10T19:01:42.326245+0000","last_became_peered":"2024-09-10T19:01:42.326245+0000","last_unstale":"2024-09-10T19:01:52.614429+0000","last_undegraded":"2024-09-10T19:01:52.614429+0000","last_fullsized":"2024-09-10T19:01:52.614429+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:31:45.800699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614981+0000","last_change":"2024-09-10T19:01:40.317340+0000","last_active":"2024-09-10T19:01:52.614981+0000","last_peered":"2024-09-10T19:01:52.614981+0000","last_clean":"2024-09-10T19:01:52.614981+0000","last_became_active":"2024-09-10T19:01:40.317088+0000","last_became_peered":"2024-09-10T19:01:40.317088+0000","last_unstale":"2024-09-10T19:01:52.614981+0000","last_undegraded":"2024-09-10T19:01:52.614981+0000","last_fullsized":"2024-09-10T19:01:52.614981+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:46:56.608265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"59'8","reported_seq":42,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:58.778807+0000","last_change":"2024-09-10T19:01:44.343307+0000","last_active":"2024-09-10T19:02:58.778807+0000","last_peered":"2024-09-10T19:02:58.778807+0000","last_clean":"2024-09-10T19:02:58.778807+0000","last_became_active":"2024-09-10T19:01:44.342984+0000","last_became_peered":"2024-09-10T19:01:44.342984+0000","last_unstale":"2024-09-10T19:02:58.778807+0000","last_undegraded":"2024-09-10T19:02:58.778807+0000","last_fullsized":"2024-09-10T19:02:58.778807+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:51:22.024642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335605+0000","last_change":"2024-09-10T19:01:46.346173+0000","last_active":"2024-09-10T19:01:51.335605+0000","last_peered":"2024-09-10T19:01:51.335605+0000","last_clean":"2024-09-10T19:01:51.335605+0000","last_became_active":"2024-09-10T19:01:46.345815+0000","last_became_peered":"2024-09-10T19:01:46.345815+0000","last_unstale":"2024-09-10T19:01:51.335605+0000","last_undegraded":"2024-09-10T19:01:51.335605+0000","last_fullsized":"2024-09-10T19:01:51.335605+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:59:56.757802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"59'28","reported_seq":81,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:58.778833+0000","last_change":"2024-09-10T19:01:42.331262+0000","last_active":"2024-09-10T19:02:58.778833+0000","last_peered":"2024-09-10T19:02:58.778833+0000","last_clean":"2024-09-10T19:02:58.778833+0000","last_became_active":"2024-09-10T19:01:42.330911+0000","last_became_peered":"2024-09-10T19:01:42.330911+0000","last_unstale":"2024-09-10T19:02:58.778833+0000","last_undegraded":"2024-09-10T19:02:58.778833+0000","last_fullsized":"2024-09-10T19:02:58.778833+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:08:58.322841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346938+0000","last_change":"2024-09-10T19:01:40.315419+0000","last_active":"2024-09-10T19:01:52.346938+0000","last_peered":"2024-09-10T19:01:52.346938+0000","last_clean":"2024-09-10T19:01:52.346938+0000","last_became_active":"2024-09-10T19:01:40.315161+0000","last_became_peered":"2024-09-10T19:01:40.315161+0000","last_unstale":"2024-09-10T19:01:52.346938+0000","last_undegraded":"2024-09-10T19:01:52.346938+0000","last_fullsized":"2024-09-10T19:01:52.346938+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:51:22.926469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336283+0000","last_change":"2024-09-10T19:01:44.344639+0000","last_active":"2024-09-10T19:01:51.336283+0000","last_peered":"2024-09-10T19:01:51.336283+0000","last_clean":"2024-09-10T19:01:51.336283+0000","last_became_active":"2024-09-10T19:01:44.344352+0000","last_became_peered":"2024-09-10T19:01:44.344352+0000","last_unstale":"2024-09-10T19:01:51.336283+0000","last_undegraded":"2024-09-10T19:01:51.336283+0000","last_fullsized":"2024-09-10T19:01:51.336283+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:26:08.244949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977330+0000","last_change":"2024-09-10T19:01:46.347861+0000","last_active":"2024-09-10T19:01:51.977330+0000","last_peered":"2024-09-10T19:01:51.977330+0000","last_clean":"2024-09-10T19:01:51.977330+0000","last_became_active":"2024-09-10T19:01:46.345916+0000","last_became_peered":"2024-09-10T19:01:46.345916+0000","last_unstale":"2024-09-10T19:01:51.977330+0000","last_undegraded":"2024-09-10T19:01:51.977330+0000","last_fullsized":"2024-09-10T19:01:51.977330+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:45:57.183548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"59'13","reported_seq":43,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.615102+0000","last_change":"2024-09-10T19:01:42.330224+0000","last_active":"2024-09-10T19:01:52.615102+0000","last_peered":"2024-09-10T19:01:52.615102+0000","last_clean":"2024-09-10T19:01:52.615102+0000","last_became_active":"2024-09-10T19:01:42.329948+0000","last_became_peered":"2024-09-10T19:01:42.329948+0000","last_unstale":"2024-09-10T19:01:52.615102+0000","last_undegraded":"2024-09-10T19:01:52.615102+0000","last_fullsized":"2024-09-10T19:01:52.615102+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:46:08.077304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"60'5","reported_seq":107,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:03:06.759329+0000","last_change":"2024-09-10T19:01:44.339700+0000","last_active":"2024-09-10T19:03:06.759329+0000","last_peered":"2024-09-10T19:03:06.759329+0000","last_clean":"2024-09-10T19:03:06.759329+0000","last_became_active":"2024-09-10T19:01:40.310716+0000","last_became_peered":"2024-09-10T19:01:40.310716+0000","last_unstale":"2024-09-10T19:03:06.759329+0000","last_undegraded":"2024-09-10T19:03:06.759329+0000","last_fullsized":"2024-09-10T19:03:06.759329+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:52:23.632309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033453020000000001,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":81,"num_read_kb":76,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.334861+0000","last_change":"2024-09-10T19:01:40.310863+0000","last_active":"2024-09-10T19:01:51.334861+0000","last_peered":"2024-09-10T19:01:51.334861+0000","last_clean":"2024-09-10T19:01:51.334861+0000","last_became_active":"2024-09-10T19:01:40.310551+0000","last_became_peered":"2024-09-10T19:01:40.310551+0000","last_unstale":"2024-09-10T19:01:51.334861+0000","last_undegraded":"2024-09-10T19:01:51.334861+0000","last_fullsized":"2024-09-10T19:01:51.334861+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:51:34.081310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335227+0000","last_change":"2024-09-10T19:01:44.341197+0000","last_active":"2024-09-10T19:01:51.335227+0000","last_peered":"2024-09-10T19:01:51.335227+0000","last_clean":"2024-09-10T19:01:51.335227+0000","last_became_active":"2024-09-10T19:01:44.340361+0000","last_became_peered":"2024-09-10T19:01:44.340361+0000","last_unstale":"2024-09-10T19:01:51.335227+0000","last_undegraded":"2024-09-10T19:01:51.335227+0000","last_fullsized":"2024-09-10T19:01:51.335227+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:38:25.636722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474539+0000","last_change":"2024-09-10T19:01:46.614318+0000","last_active":"2024-09-10T19:01:52.474539+0000","last_peered":"2024-09-10T19:01:52.474539+0000","last_clean":"2024-09-10T19:01:52.474539+0000","last_became_active":"2024-09-10T19:01:46.614022+0000","last_became_peered":"2024-09-10T19:01:46.614022+0000","last_unstale":"2024-09-10T19:01:52.474539+0000","last_undegraded":"2024-09-10T19:01:52.474539+0000","last_fullsized":"2024-09-10T19:01:52.474539+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:40:01.399069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"59'12","reported_seq":34,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336634+0000","last_change":"2024-09-10T19:01:42.328339+0000","last_active":"2024-09-10T19:01:51.336634+0000","last_peered":"2024-09-10T19:01:51.336634+0000","last_clean":"2024-09-10T19:01:51.336634+0000","last_became_active":"2024-09-10T19:01:42.328027+0000","last_became_peered":"2024-09-10T19:01:42.328027+0000","last_unstale":"2024-09-10T19:01:51.336634+0000","last_undegraded":"2024-09-10T19:01:51.336634+0000","last_fullsized":"2024-09-10T19:01:51.336634+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:11:20.315951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"54'1","reported_seq":24,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336081+0000","last_change":"2024-09-10T19:01:44.331881+0000","last_active":"2024-09-10T19:01:51.336081+0000","last_peered":"2024-09-10T19:01:51.336081+0000","last_clean":"2024-09-10T19:01:51.336081+0000","last_became_active":"2024-09-10T19:01:40.310783+0000","last_became_peered":"2024-09-10T19:01:40.310783+0000","last_unstale":"2024-09-10T19:01:51.336081+0000","last_undegraded":"2024-09-10T19:01:51.336081+0000","last_fullsized":"2024-09-10T19:01:51.336081+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:16:45.275300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043770099999999998,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474452+0000","last_change":"2024-09-10T19:01:40.321999+0000","last_active":"2024-09-10T19:01:52.474452+0000","last_peered":"2024-09-10T19:01:52.474452+0000","last_clean":"2024-09-10T19:01:52.474452+0000","last_became_active":"2024-09-10T19:01:40.321349+0000","last_became_peered":"2024-09-10T19:01:40.321349+0000","last_unstale":"2024-09-10T19:01:52.474452+0000","last_undegraded":"2024-09-10T19:01:52.474452+0000","last_fullsized":"2024-09-10T19:01:52.474452+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:46:19.376593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346684+0000","last_change":"2024-09-10T19:01:44.344941+0000","last_active":"2024-09-10T19:01:52.346684+0000","last_peered":"2024-09-10T19:01:52.346684+0000","last_clean":"2024-09-10T19:01:52.346684+0000","last_became_active":"2024-09-10T19:01:44.344615+0000","last_became_peered":"2024-09-10T19:01:44.344615+0000","last_unstale":"2024-09-10T19:01:52.346684+0000","last_undegraded":"2024-09-10T19:01:52.346684+0000","last_fullsized":"2024-09-10T19:01:52.346684+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:25:34.432176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614751+0000","last_change":"2024-09-10T19:01:46.346738+0000","last_active":"2024-09-10T19:01:52.614751+0000","last_peered":"2024-09-10T19:01:52.614751+0000","last_clean":"2024-09-10T19:01:52.614751+0000","last_became_active":"2024-09-10T19:01:46.346475+0000","last_became_peered":"2024-09-10T19:01:46.346475+0000","last_unstale":"2024-09-10T19:01:52.614751+0000","last_undegraded":"2024-09-10T19:01:52.614751+0000","last_fullsized":"2024-09-10T19:01:52.614751+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:32:21.180179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"59'14","reported_seq":38,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.608369+0000","last_change":"2024-09-10T19:01:42.380619+0000","last_active":"2024-09-10T19:01:52.608369+0000","last_peered":"2024-09-10T19:01:52.608369+0000","last_clean":"2024-09-10T19:01:52.608369+0000","last_became_active":"2024-09-10T19:01:42.380306+0000","last_became_peered":"2024-09-10T19:01:42.380306+0000","last_unstale":"2024-09-10T19:01:52.608369+0000","last_undegraded":"2024-09-10T19:01:52.608369+0000","last_fullsized":"2024-09-10T19:01:52.608369+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:54:18.570288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"56'2","reported_seq":25,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977002+0000","last_change":"2024-09-10T19:01:44.339090+0000","last_active":"2024-09-10T19:01:51.977002+0000","last_peered":"2024-09-10T19:01:51.977002+0000","last_clean":"2024-09-10T19:01:51.977002+0000","last_became_active":"2024-09-10T19:01:40.313472+0000","last_became_peered":"2024-09-10T19:01:40.313472+0000","last_unstale":"2024-09-10T19:01:51.977002+0000","last_undegraded":"2024-09-10T19:01:51.977002+0000","last_fullsized":"2024-09-10T19:01:51.977002+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:52:25.173129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019757860000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346883+0000","last_change":"2024-09-10T19:01:40.319837+0000","last_active":"2024-09-10T19:01:52.346883+0000","last_peered":"2024-09-10T19:01:52.346883+0000","last_clean":"2024-09-10T19:01:52.346883+0000","last_became_active":"2024-09-10T19:01:40.319316+0000","last_became_peered":"2024-09-10T19:01:40.319316+0000","last_unstale":"2024-09-10T19:01:52.346883+0000","last_undegraded":"2024-09-10T19:01:52.346883+0000","last_fullsized":"2024-09-10T19:01:52.346883+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:48:30.471262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"61'79","reported_seq":52,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:19.076563+0000","last_change":"2024-09-10T19:01:26.169270+0000","last_active":"2024-09-10T19:02:19.076563+0000","last_peered":"2024-09-10T19:02:19.076563+0000","last_clean":"2024-09-10T19:02:19.076563+0000","last_became_active":"2024-09-10T19:01:26.161939+0000","last_became_peered":"2024-09-10T19:01:26.161939+0000","last_unstale":"2024-09-10T19:02:19.076563+0000","last_undegraded":"2024-09-10T19:02:19.076563+0000","last_fullsized":"2024-09-10T19:02:19.076563+0000","mapping_epoch":49,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":50,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T18:59:30.525956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T18:59:30.525956+0000","last_clean_scrub_stamp":"2024-09-10T18:59:30.525956+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:57:46.815060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.473602+0000","last_change":"2024-09-10T19:01:44.342189+0000","last_active":"2024-09-10T19:01:52.473602+0000","last_peered":"2024-09-10T19:01:52.473602+0000","last_clean":"2024-09-10T19:01:52.473602+0000","last_became_active":"2024-09-10T19:01:44.341900+0000","last_became_peered":"2024-09-10T19:01:44.341900+0000","last_unstale":"2024-09-10T19:01:52.473602+0000","last_undegraded":"2024-09-10T19:01:52.473602+0000","last_fullsized":"2024-09-10T19:01:52.473602+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:26:39.302809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.347209+0000","last_change":"2024-09-10T19:01:46.350066+0000","last_active":"2024-09-10T19:01:52.347209+0000","last_peered":"2024-09-10T19:01:52.347209+0000","last_clean":"2024-09-10T19:01:52.347209+0000","last_became_active":"2024-09-10T19:01:46.349776+0000","last_became_peered":"2024-09-10T19:01:46.349776+0000","last_unstale":"2024-09-10T19:01:52.347209+0000","last_undegraded":"2024-09-10T19:01:52.347209+0000","last_fullsized":"2024-09-10T19:01:52.347209+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:26:27.446823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"59'19","reported_seq":47,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.607674+0000","last_change":"2024-09-10T19:01:42.379603+0000","last_active":"2024-09-10T19:01:52.607674+0000","last_peered":"2024-09-10T19:01:52.607674+0000","last_clean":"2024-09-10T19:01:52.607674+0000","last_became_active":"2024-09-10T19:01:42.378718+0000","last_became_peered":"2024-09-10T19:01:42.378718+0000","last_unstale":"2024-09-10T19:01:52.607674+0000","last_undegraded":"2024-09-10T19:01:52.607674+0000","last_fullsized":"2024-09-10T19:01:52.607674+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:24:21.723058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335307+0000","last_change":"2024-09-10T19:01:40.312962+0000","last_active":"2024-09-10T19:01:51.335307+0000","last_peered":"2024-09-10T19:01:51.335307+0000","last_clean":"2024-09-10T19:01:51.335307+0000","last_became_active":"2024-09-10T19:01:40.312720+0000","last_became_peered":"2024-09-10T19:01:40.312720+0000","last_unstale":"2024-09-10T19:01:51.335307+0000","last_undegraded":"2024-09-10T19:01:51.335307+0000","last_fullsized":"2024-09-10T19:01:51.335307+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:00:08.645352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.334746+0000","last_change":"2024-09-10T19:01:44.340037+0000","last_active":"2024-09-10T19:01:51.334746+0000","last_peered":"2024-09-10T19:01:51.334746+0000","last_clean":"2024-09-10T19:01:51.334746+0000","last_became_active":"2024-09-10T19:01:44.339699+0000","last_became_peered":"2024-09-10T19:01:44.339699+0000","last_unstale":"2024-09-10T19:01:51.334746+0000","last_undegraded":"2024-09-10T19:01:51.334746+0000","last_fullsized":"2024-09-10T19:01:51.334746+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:16:51.938721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.473990+0000","last_change":"2024-09-10T19:01:46.347613+0000","last_active":"2024-09-10T19:01:52.473990+0000","last_peered":"2024-09-10T19:01:52.473990+0000","last_clean":"2024-09-10T19:01:52.473990+0000","last_became_active":"2024-09-10T19:01:46.347345+0000","last_became_peered":"2024-09-10T19:01:46.347345+0000","last_unstale":"2024-09-10T19:01:52.473990+0000","last_undegraded":"2024-09-10T19:01:52.473990+0000","last_fullsized":"2024-09-10T19:01:52.473990+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:27:43.490758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"59'9","reported_seq":32,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336296+0000","last_change":"2024-09-10T19:01:42.325200+0000","last_active":"2024-09-10T19:01:51.336296+0000","last_peered":"2024-09-10T19:01:51.336296+0000","last_clean":"2024-09-10T19:01:51.336296+0000","last_became_active":"2024-09-10T19:01:42.324853+0000","last_became_peered":"2024-09-10T19:01:42.324853+0000","last_unstale":"2024-09-10T19:01:51.336296+0000","last_undegraded":"2024-09-10T19:01:51.336296+0000","last_fullsized":"2024-09-10T19:01:51.336296+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:46:53.265923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614900+0000","last_change":"2024-09-10T19:01:40.308532+0000","last_active":"2024-09-10T19:01:52.614900+0000","last_peered":"2024-09-10T19:01:52.614900+0000","last_clean":"2024-09-10T19:01:52.614900+0000","last_became_active":"2024-09-10T19:01:40.308261+0000","last_became_peered":"2024-09-10T19:01:40.308261+0000","last_unstale":"2024-09-10T19:01:52.614900+0000","last_undegraded":"2024-09-10T19:01:52.614900+0000","last_fullsized":"2024-09-10T19:01:52.614900+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:53:24.762541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.334722+0000","last_change":"2024-09-10T19:01:44.343461+0000","last_active":"2024-09-10T19:01:51.334722+0000","last_peered":"2024-09-10T19:01:51.334722+0000","last_clean":"2024-09-10T19:01:51.334722+0000","last_became_active":"2024-09-10T19:01:44.342749+0000","last_became_peered":"2024-09-10T19:01:44.342749+0000","last_unstale":"2024-09-10T19:01:51.334722+0000","last_undegraded":"2024-09-10T19:01:51.334722+0000","last_fullsized":"2024-09-10T19:01:51.334722+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:02:53.728630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335889+0000","last_change":"2024-09-10T19:01:46.346711+0000","last_active":"2024-09-10T19:01:51.335889+0000","last_peered":"2024-09-10T19:01:51.335889+0000","last_clean":"2024-09-10T19:01:51.335889+0000","last_became_active":"2024-09-10T19:01:46.346422+0000","last_became_peered":"2024-09-10T19:01:46.346422+0000","last_unstale":"2024-09-10T19:01:51.335889+0000","last_undegraded":"2024-09-10T19:01:51.335889+0000","last_fullsized":"2024-09-10T19:01:51.335889+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:24:21.410686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"59'15","reported_seq":41,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346592+0000","last_change":"2024-09-10T19:01:42.379088+0000","last_active":"2024-09-10T19:01:52.346592+0000","last_peered":"2024-09-10T19:01:52.346592+0000","last_clean":"2024-09-10T19:01:52.346592+0000","last_became_active":"2024-09-10T19:01:42.378763+0000","last_became_peered":"2024-09-10T19:01:42.378763+0000","last_unstale":"2024-09-10T19:01:52.346592+0000","last_undegraded":"2024-09-10T19:01:52.346592+0000","last_fullsized":"2024-09-10T19:01:52.346592+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:49:33.777985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335296+0000","last_change":"2024-09-10T19:01:40.313052+0000","last_active":"2024-09-10T19:01:51.335296+0000","last_peered":"2024-09-10T19:01:51.335296+0000","last_clean":"2024-09-10T19:01:51.335296+0000","last_became_active":"2024-09-10T19:01:40.312787+0000","last_became_peered":"2024-09-10T19:01:40.312787+0000","last_unstale":"2024-09-10T19:01:51.335296+0000","last_undegraded":"2024-09-10T19:01:51.335296+0000","last_fullsized":"2024-09-10T19:01:51.335296+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:11:55.269886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"59'8","reported_seq":39,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:58.778965+0000","last_change":"2024-09-10T19:01:44.344555+0000","last_active":"2024-09-10T19:02:58.778965+0000","last_peered":"2024-09-10T19:02:58.778965+0000","last_clean":"2024-09-10T19:02:58.778965+0000","last_became_active":"2024-09-10T19:01:44.344303+0000","last_became_peered":"2024-09-10T19:01:44.344303+0000","last_unstale":"2024-09-10T19:02:58.778965+0000","last_undegraded":"2024-09-10T19:02:58.778965+0000","last_fullsized":"2024-09-10T19:02:58.778965+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:34:04.681831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346648+0000","last_change":"2024-09-10T19:01:46.615644+0000","last_active":"2024-09-10T19:01:52.346648+0000","last_peered":"2024-09-10T19:01:52.346648+0000","last_clean":"2024-09-10T19:01:52.346648+0000","last_became_active":"2024-09-10T19:01:46.614542+0000","last_became_peered":"2024-09-10T19:01:46.614542+0000","last_unstale":"2024-09-10T19:01:52.346648+0000","last_undegraded":"2024-09-10T19:01:52.346648+0000","last_fullsized":"2024-09-10T19:01:52.346648+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:53:22.911167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"59'12","reported_seq":39,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330103+0000","last_change":"2024-09-10T19:01:42.327921+0000","last_active":"2024-09-10T19:01:51.330103+0000","last_peered":"2024-09-10T19:01:51.330103+0000","last_clean":"2024-09-10T19:01:51.330103+0000","last_became_active":"2024-09-10T19:01:42.327569+0000","last_became_peered":"2024-09-10T19:01:42.327569+0000","last_unstale":"2024-09-10T19:01:51.330103+0000","last_undegraded":"2024-09-10T19:01:51.330103+0000","last_fullsized":"2024-09-10T19:01:51.330103+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:45:30.416835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"52'2","reported_seq":35,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330681+0000","last_change":"2024-09-10T19:01:40.313889+0000","last_active":"2024-09-10T19:01:51.330681+0000","last_peered":"2024-09-10T19:01:51.330681+0000","last_clean":"2024-09-10T19:01:51.330681+0000","last_became_active":"2024-09-10T19:01:40.313577+0000","last_became_peered":"2024-09-10T19:01:40.313577+0000","last_unstale":"2024-09-10T19:01:51.330681+0000","last_undegraded":"2024-09-10T19:01:51.330681+0000","last_fullsized":"2024-09-10T19:01:51.330681+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:28:09.705498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.334753+0000","last_change":"2024-09-10T19:01:44.340996+0000","last_active":"2024-09-10T19:01:51.334753+0000","last_peered":"2024-09-10T19:01:51.334753+0000","last_clean":"2024-09-10T19:01:51.334753+0000","last_became_active":"2024-09-10T19:01:44.340642+0000","last_became_peered":"2024-09-10T19:01:44.340642+0000","last_unstale":"2024-09-10T19:01:51.334753+0000","last_undegraded":"2024-09-10T19:01:51.334753+0000","last_fullsized":"2024-09-10T19:01:51.334753+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:58:27.613052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976778+0000","last_change":"2024-09-10T19:01:46.347759+0000","last_active":"2024-09-10T19:01:51.976778+0000","last_peered":"2024-09-10T19:01:51.976778+0000","last_clean":"2024-09-10T19:01:51.976778+0000","last_became_active":"2024-09-10T19:01:46.345502+0000","last_became_peered":"2024-09-10T19:01:46.345502+0000","last_unstale":"2024-09-10T19:01:51.976778+0000","last_undegraded":"2024-09-10T19:01:51.976778+0000","last_fullsized":"2024-09-10T19:01:51.976778+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:04:31.584396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"59'9","reported_seq":32,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336578+0000","last_change":"2024-09-10T19:01:42.379173+0000","last_active":"2024-09-10T19:01:51.336578+0000","last_peered":"2024-09-10T19:01:51.336578+0000","last_clean":"2024-09-10T19:01:51.336578+0000","last_became_active":"2024-09-10T19:01:42.378690+0000","last_became_peered":"2024-09-10T19:01:42.378690+0000","last_unstale":"2024-09-10T19:01:51.336578+0000","last_undegraded":"2024-09-10T19:01:51.336578+0000","last_fullsized":"2024-09-10T19:01:51.336578+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:49:32.120315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"52'1","reported_seq":27,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.334862+0000","last_change":"2024-09-10T19:01:40.311489+0000","last_active":"2024-09-10T19:01:51.334862+0000","last_peered":"2024-09-10T19:01:51.334862+0000","last_clean":"2024-09-10T19:01:51.334862+0000","last_became_active":"2024-09-10T19:01:40.311214+0000","last_became_peered":"2024-09-10T19:01:40.311214+0000","last_unstale":"2024-09-10T19:01:51.334862+0000","last_undegraded":"2024-09-10T19:01:51.334862+0000","last_fullsized":"2024-09-10T19:01:51.334862+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:15:11.232578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":8,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977495+0000","last_change":"2024-09-10T19:01:44.347275+0000","last_active":"2024-09-10T19:01:51.977495+0000","last_peered":"2024-09-10T19:01:51.977495+0000","last_clean":"2024-09-10T19:01:51.977495+0000","last_became_active":"2024-09-10T19:01:44.346984+0000","last_became_peered":"2024-09-10T19:01:44.346984+0000","last_unstale":"2024-09-10T19:01:51.977495+0000","last_undegraded":"2024-09-10T19:01:51.977495+0000","last_fullsized":"2024-09-10T19:01:51.977495+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:34:01.997058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335142+0000","last_change":"2024-09-10T19:01:46.346649+0000","last_active":"2024-09-10T19:01:51.335142+0000","last_peered":"2024-09-10T19:01:51.335142+0000","last_clean":"2024-09-10T19:01:51.335142+0000","last_became_active":"2024-09-10T19:01:46.346381+0000","last_became_peered":"2024-09-10T19:01:46.346381+0000","last_unstale":"2024-09-10T19:01:51.335142+0000","last_undegraded":"2024-09-10T19:01:51.335142+0000","last_fullsized":"2024-09-10T19:01:51.335142+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:24:14.210943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.607750+0000","last_change":"2024-09-10T19:01:40.312738+0000","last_active":"2024-09-10T19:01:52.607750+0000","last_peered":"2024-09-10T19:01:52.607750+0000","last_clean":"2024-09-10T19:01:52.607750+0000","last_became_active":"2024-09-10T19:01:40.312196+0000","last_became_peered":"2024-09-10T19:01:40.312196+0000","last_unstale":"2024-09-10T19:01:52.607750+0000","last_undegraded":"2024-09-10T19:01:52.607750+0000","last_fullsized":"2024-09-10T19:01:52.607750+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:18:29.748034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"59'6","reported_seq":25,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977548+0000","last_change":"2024-09-10T19:01:42.378946+0000","last_active":"2024-09-10T19:01:51.977548+0000","last_peered":"2024-09-10T19:01:51.977548+0000","last_clean":"2024-09-10T19:01:51.977548+0000","last_became_active":"2024-09-10T19:01:42.378693+0000","last_became_peered":"2024-09-10T19:01:42.378693+0000","last_unstale":"2024-09-10T19:01:51.977548+0000","last_undegraded":"2024-09-10T19:01:51.977548+0000","last_fullsized":"2024-09-10T19:01:51.977548+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:51:36.571214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346467+0000","last_change":"2024-09-10T19:01:44.343221+0000","last_active":"2024-09-10T19:01:52.346467+0000","last_peered":"2024-09-10T19:01:52.346467+0000","last_clean":"2024-09-10T19:01:52.346467+0000","last_became_active":"2024-09-10T19:01:44.342869+0000","last_became_peered":"2024-09-10T19:01:44.342869+0000","last_unstale":"2024-09-10T19:01:52.346467+0000","last_undegraded":"2024-09-10T19:01:52.346467+0000","last_fullsized":"2024-09-10T19:01:52.346467+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:38:26.130461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474546+0000","last_change":"2024-09-10T19:01:46.614487+0000","last_active":"2024-09-10T19:01:52.474546+0000","last_peered":"2024-09-10T19:01:52.474546+0000","last_clean":"2024-09-10T19:01:52.474546+0000","last_became_active":"2024-09-10T19:01:46.614229+0000","last_became_peered":"2024-09-10T19:01:46.614229+0000","last_unstale":"2024-09-10T19:01:52.474546+0000","last_undegraded":"2024-09-10T19:01:52.474546+0000","last_fullsized":"2024-09-10T19:01:52.474546+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:22.904465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"59'10","reported_seq":31,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977219+0000","last_change":"2024-09-10T19:01:42.378894+0000","last_active":"2024-09-10T19:01:51.977219+0000","last_peered":"2024-09-10T19:01:51.977219+0000","last_clean":"2024-09-10T19:01:51.977219+0000","last_became_active":"2024-09-10T19:01:42.378618+0000","last_became_peered":"2024-09-10T19:01:42.378618+0000","last_unstale":"2024-09-10T19:01:51.977219+0000","last_undegraded":"2024-09-10T19:01:51.977219+0000","last_fullsized":"2024-09-10T19:01:51.977219+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:52:35.762940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346955+0000","last_change":"2024-09-10T19:01:40.319717+0000","last_active":"2024-09-10T19:01:52.346955+0000","last_peered":"2024-09-10T19:01:52.346955+0000","last_clean":"2024-09-10T19:01:52.346955+0000","last_became_active":"2024-09-10T19:01:40.319447+0000","last_became_peered":"2024-09-10T19:01:40.319447+0000","last_unstale":"2024-09-10T19:01:52.346955+0000","last_undegraded":"2024-09-10T19:01:52.346955+0000","last_fullsized":"2024-09-10T19:01:52.346955+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:55.702268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"59'8","reported_seq":39,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:58.778965+0000","last_change":"2024-09-10T19:01:44.344270+0000","last_active":"2024-09-10T19:02:58.778965+0000","last_peered":"2024-09-10T19:02:58.778965+0000","last_clean":"2024-09-10T19:02:58.778965+0000","last_became_active":"2024-09-10T19:01:44.344000+0000","last_became_peered":"2024-09-10T19:01:44.344000+0000","last_unstale":"2024-09-10T19:02:58.778965+0000","last_undegraded":"2024-09-10T19:02:58.778965+0000","last_fullsized":"2024-09-10T19:02:58.778965+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:04:05.225998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"59'1","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336363+0000","last_change":"2024-09-10T19:01:46.347577+0000","last_active":"2024-09-10T19:01:51.336363+0000","last_peered":"2024-09-10T19:01:51.336363+0000","last_clean":"2024-09-10T19:01:51.336363+0000","last_became_active":"2024-09-10T19:01:46.347293+0000","last_became_peered":"2024-09-10T19:01:46.347293+0000","last_unstale":"2024-09-10T19:01:51.336363+0000","last_undegraded":"2024-09-10T19:01:51.336363+0000","last_fullsized":"2024-09-10T19:01:51.336363+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:12:23.785867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"59'9","reported_seq":32,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.347497+0000","last_change":"2024-09-10T19:01:42.380408+0000","last_active":"2024-09-10T19:01:52.347497+0000","last_peered":"2024-09-10T19:01:52.347497+0000","last_clean":"2024-09-10T19:01:52.347497+0000","last_became_active":"2024-09-10T19:01:42.380069+0000","last_became_peered":"2024-09-10T19:01:42.380069+0000","last_unstale":"2024-09-10T19:01:52.347497+0000","last_undegraded":"2024-09-10T19:01:52.347497+0000","last_fullsized":"2024-09-10T19:01:52.347497+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:55:22.417952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336538+0000","last_change":"2024-09-10T19:01:40.315253+0000","last_active":"2024-09-10T19:01:51.336538+0000","last_peered":"2024-09-10T19:01:51.336538+0000","last_clean":"2024-09-10T19:01:51.336538+0000","last_became_active":"2024-09-10T19:01:40.314605+0000","last_became_peered":"2024-09-10T19:01:40.314605+0000","last_unstale":"2024-09-10T19:01:51.336538+0000","last_undegraded":"2024-09-10T19:01:51.336538+0000","last_fullsized":"2024-09-10T19:01:51.336538+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:53:30.768785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"59'8","reported_seq":39,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:58.779002+0000","last_change":"2024-09-10T19:01:44.348398+0000","last_active":"2024-09-10T19:02:58.779002+0000","last_peered":"2024-09-10T19:02:58.779002+0000","last_clean":"2024-09-10T19:02:58.779002+0000","last_became_active":"2024-09-10T19:01:44.347356+0000","last_became_peered":"2024-09-10T19:01:44.347356+0000","last_unstale":"2024-09-10T19:02:58.779002+0000","last_undegraded":"2024-09-10T19:02:58.779002+0000","last_fullsized":"2024-09-10T19:02:58.779002+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:28:30.165391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330510+0000","last_change":"2024-09-10T19:01:46.348010+0000","last_active":"2024-09-10T19:01:51.330510+0000","last_peered":"2024-09-10T19:01:51.330510+0000","last_clean":"2024-09-10T19:01:51.330510+0000","last_became_active":"2024-09-10T19:01:46.346937+0000","last_became_peered":"2024-09-10T19:01:46.346937+0000","last_unstale":"2024-09-10T19:01:51.330510+0000","last_undegraded":"2024-09-10T19:01:51.330510+0000","last_fullsized":"2024-09-10T19:01:51.330510+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:06:32.101879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"59'9","reported_seq":32,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614355+0000","last_change":"2024-09-10T19:01:42.328462+0000","last_active":"2024-09-10T19:01:52.614355+0000","last_peered":"2024-09-10T19:01:52.614355+0000","last_clean":"2024-09-10T19:01:52.614355+0000","last_became_active":"2024-09-10T19:01:42.327579+0000","last_became_peered":"2024-09-10T19:01:42.327579+0000","last_unstale":"2024-09-10T19:01:52.614355+0000","last_undegraded":"2024-09-10T19:01:52.614355+0000","last_fullsized":"2024-09-10T19:01:52.614355+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:51:59.209933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.608466+0000","last_change":"2024-09-10T19:01:40.317780+0000","last_active":"2024-09-10T19:01:52.608466+0000","last_peered":"2024-09-10T19:01:52.608466+0000","last_clean":"2024-09-10T19:01:52.608466+0000","last_became_active":"2024-09-10T19:01:40.317346+0000","last_became_peered":"2024-09-10T19:01:40.317346+0000","last_unstale":"2024-09-10T19:01:52.608466+0000","last_undegraded":"2024-09-10T19:01:52.608466+0000","last_fullsized":"2024-09-10T19:01:52.608466+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:30:10.923626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976787+0000","last_change":"2024-09-10T19:01:44.347956+0000","last_active":"2024-09-10T19:01:51.976787+0000","last_peered":"2024-09-10T19:01:51.976787+0000","last_clean":"2024-09-10T19:01:51.976787+0000","last_became_active":"2024-09-10T19:01:44.347694+0000","last_became_peered":"2024-09-10T19:01:44.347694+0000","last_unstale":"2024-09-10T19:01:51.976787+0000","last_undegraded":"2024-09-10T19:01:51.976787+0000","last_fullsized":"2024-09-10T19:01:51.976787+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:09:18.783003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335556+0000","last_change":"2024-09-10T19:01:46.347348+0000","last_active":"2024-09-10T19:01:51.335556+0000","last_peered":"2024-09-10T19:01:51.335556+0000","last_clean":"2024-09-10T19:01:51.335556+0000","last_became_active":"2024-09-10T19:01:46.347051+0000","last_became_peered":"2024-09-10T19:01:46.347051+0000","last_unstale":"2024-09-10T19:01:51.335556+0000","last_undegraded":"2024-09-10T19:01:51.335556+0000","last_fullsized":"2024-09-10T19:01:51.335556+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:12:12.230555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"59'11","reported_seq":35,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330313+0000","last_change":"2024-09-10T19:01:42.325262+0000","last_active":"2024-09-10T19:01:51.330313+0000","last_peered":"2024-09-10T19:01:51.330313+0000","last_clean":"2024-09-10T19:01:51.330313+0000","last_became_active":"2024-09-10T19:01:42.324987+0000","last_became_peered":"2024-09-10T19:01:42.324987+0000","last_unstale":"2024-09-10T19:01:51.330313+0000","last_undegraded":"2024-09-10T19:01:51.330313+0000","last_fullsized":"2024-09-10T19:01:51.330313+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:52:03.736308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614862+0000","last_change":"2024-09-10T19:01:40.310304+0000","last_active":"2024-09-10T19:01:52.614862+0000","last_peered":"2024-09-10T19:01:52.614862+0000","last_clean":"2024-09-10T19:01:52.614862+0000","last_became_active":"2024-09-10T19:01:40.310062+0000","last_became_peered":"2024-09-10T19:01:40.310062+0000","last_unstale":"2024-09-10T19:01:52.614862+0000","last_undegraded":"2024-09-10T19:01:52.614862+0000","last_fullsized":"2024-09-10T19:01:52.614862+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:50:16.846872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.613367+0000","last_change":"2024-09-10T19:01:44.347398+0000","last_active":"2024-09-10T19:01:52.613367+0000","last_peered":"2024-09-10T19:01:52.613367+0000","last_clean":"2024-09-10T19:01:52.613367+0000","last_became_active":"2024-09-10T19:01:44.347105+0000","last_became_peered":"2024-09-10T19:01:44.347105+0000","last_unstale":"2024-09-10T19:01:52.613367+0000","last_undegraded":"2024-09-10T19:01:52.613367+0000","last_fullsized":"2024-09-10T19:01:52.613367+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:17:42.033789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976518+0000","last_change":"2024-09-10T19:01:46.343744+0000","last_active":"2024-09-10T19:01:51.976518+0000","last_peered":"2024-09-10T19:01:51.976518+0000","last_clean":"2024-09-10T19:01:51.976518+0000","last_became_active":"2024-09-10T19:01:46.343467+0000","last_became_peered":"2024-09-10T19:01:46.343467+0000","last_unstale":"2024-09-10T19:01:51.976518+0000","last_undegraded":"2024-09-10T19:01:51.976518+0000","last_fullsized":"2024-09-10T19:01:51.976518+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:19:50.625620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"59'4","reported_seq":22,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976679+0000","last_change":"2024-09-10T19:01:42.329213+0000","last_active":"2024-09-10T19:01:51.976679+0000","last_peered":"2024-09-10T19:01:51.976679+0000","last_clean":"2024-09-10T19:01:51.976679+0000","last_became_active":"2024-09-10T19:01:42.328924+0000","last_became_peered":"2024-09-10T19:01:42.328924+0000","last_unstale":"2024-09-10T19:01:51.976679+0000","last_undegraded":"2024-09-10T19:01:51.976679+0000","last_fullsized":"2024-09-10T19:01:51.976679+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:09:00.923719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.347273+0000","last_change":"2024-09-10T19:01:40.316478+0000","last_active":"2024-09-10T19:01:52.347273+0000","last_peered":"2024-09-10T19:01:52.347273+0000","last_clean":"2024-09-10T19:01:52.347273+0000","last_became_active":"2024-09-10T19:01:40.316229+0000","last_became_peered":"2024-09-10T19:01:40.316229+0000","last_unstale":"2024-09-10T19:01:52.347273+0000","last_undegraded":"2024-09-10T19:01:52.347273+0000","last_fullsized":"2024-09-10T19:01:52.347273+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:12:40.874738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.607305+0000","last_change":"2024-09-10T19:01:44.351243+0000","last_active":"2024-09-10T19:01:52.607305+0000","last_peered":"2024-09-10T19:01:52.607305+0000","last_clean":"2024-09-10T19:01:52.607305+0000","last_became_active":"2024-09-10T19:01:44.350871+0000","last_became_peered":"2024-09-10T19:01:44.350871+0000","last_unstale":"2024-09-10T19:01:52.607305+0000","last_undegraded":"2024-09-10T19:01:52.607305+0000","last_fullsized":"2024-09-10T19:01:52.607305+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:35:25.052911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.473901+0000","last_change":"2024-09-10T19:01:46.345647+0000","last_active":"2024-09-10T19:01:52.473901+0000","last_peered":"2024-09-10T19:01:52.473901+0000","last_clean":"2024-09-10T19:01:52.473901+0000","last_became_active":"2024-09-10T19:01:46.345357+0000","last_became_peered":"2024-09-10T19:01:46.345357+0000","last_unstale":"2024-09-10T19:01:52.473901+0000","last_undegraded":"2024-09-10T19:01:52.473901+0000","last_fullsized":"2024-09-10T19:01:52.473901+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:49:09.926505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"59'11","reported_seq":35,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976679+0000","last_change":"2024-09-10T19:01:42.379639+0000","last_active":"2024-09-10T19:01:51.976679+0000","last_peered":"2024-09-10T19:01:51.976679+0000","last_clean":"2024-09-10T19:01:51.976679+0000","last_became_active":"2024-09-10T19:01:42.379006+0000","last_became_peered":"2024-09-10T19:01:42.379006+0000","last_unstale":"2024-09-10T19:01:51.976679+0000","last_undegraded":"2024-09-10T19:01:51.976679+0000","last_fullsized":"2024-09-10T19:01:51.976679+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:00:34.998663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.609048+0000","last_change":"2024-09-10T19:01:40.317573+0000","last_active":"2024-09-10T19:01:52.609048+0000","last_peered":"2024-09-10T19:01:52.609048+0000","last_clean":"2024-09-10T19:01:52.609048+0000","last_became_active":"2024-09-10T19:01:40.317278+0000","last_became_peered":"2024-09-10T19:01:40.317278+0000","last_unstale":"2024-09-10T19:01:52.609048+0000","last_undegraded":"2024-09-10T19:01:52.609048+0000","last_fullsized":"2024-09-10T19:01:52.609048+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:21:15.544228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.473246+0000","last_change":"2024-09-10T19:01:44.345955+0000","last_active":"2024-09-10T19:01:52.473246+0000","last_peered":"2024-09-10T19:01:52.473246+0000","last_clean":"2024-09-10T19:01:52.473246+0000","last_became_active":"2024-09-10T19:01:44.345674+0000","last_became_peered":"2024-09-10T19:01:44.345674+0000","last_unstale":"2024-09-10T19:01:52.473246+0000","last_undegraded":"2024-09-10T19:01:52.473246+0000","last_fullsized":"2024-09-10T19:01:52.473246+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:34:44.118992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976673+0000","last_change":"2024-09-10T19:01:46.615441+0000","last_active":"2024-09-10T19:01:51.976673+0000","last_peered":"2024-09-10T19:01:51.976673+0000","last_clean":"2024-09-10T19:01:51.976673+0000","last_became_active":"2024-09-10T19:01:46.614382+0000","last_became_peered":"2024-09-10T19:01:46.614382+0000","last_unstale":"2024-09-10T19:01:51.976673+0000","last_undegraded":"2024-09-10T19:01:51.976673+0000","last_fullsized":"2024-09-10T19:01:51.976673+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:03:28.095017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"59'1","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.474200+0000","last_change":"2024-09-10T19:01:46.347709+0000","last_active":"2024-09-10T19:01:52.474200+0000","last_peered":"2024-09-10T19:01:52.474200+0000","last_clean":"2024-09-10T19:01:52.474200+0000","last_became_active":"2024-09-10T19:01:46.347472+0000","last_became_peered":"2024-09-10T19:01:46.347472+0000","last_unstale":"2024-09-10T19:01:52.474200+0000","last_undegraded":"2024-09-10T19:01:52.474200+0000","last_fullsized":"2024-09-10T19:01:52.474200+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:07:55.759839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.346625+0000","last_change":"2024-09-10T19:01:40.319612+0000","last_active":"2024-09-10T19:01:52.346625+0000","last_peered":"2024-09-10T19:01:52.346625+0000","last_clean":"2024-09-10T19:01:52.346625+0000","last_became_active":"2024-09-10T19:01:40.319346+0000","last_became_peered":"2024-09-10T19:01:40.319346+0000","last_unstale":"2024-09-10T19:01:52.346625+0000","last_undegraded":"2024-09-10T19:01:52.346625+0000","last_fullsized":"2024-09-10T19:01:52.346625+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:14:42.443756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"59'10","reported_seq":31,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336543+0000","last_change":"2024-09-10T19:01:42.380821+0000","last_active":"2024-09-10T19:01:51.336543+0000","last_peered":"2024-09-10T19:01:51.336543+0000","last_clean":"2024-09-10T19:01:51.336543+0000","last_became_active":"2024-09-10T19:01:42.380475+0000","last_became_peered":"2024-09-10T19:01:42.380475+0000","last_unstale":"2024-09-10T19:01:51.336543+0000","last_undegraded":"2024-09-10T19:01:51.336543+0000","last_fullsized":"2024-09-10T19:01:51.336543+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:00:36.362315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"59'8","reported_seq":42,"reported_epoch":61,"state":"active+clean","last_fresh":"2024-09-10T19:02:58.778988+0000","last_change":"2024-09-10T19:01:44.352182+0000","last_active":"2024-09-10T19:02:58.778988+0000","last_peered":"2024-09-10T19:02:58.778988+0000","last_clean":"2024-09-10T19:02:58.778988+0000","last_became_active":"2024-09-10T19:01:44.351353+0000","last_became_peered":"2024-09-10T19:01:44.351353+0000","last_unstale":"2024-09-10T19:02:58.778988+0000","last_undegraded":"2024-09-10T19:02:58.778988+0000","last_fullsized":"2024-09-10T19:02:58.778988+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:14:27.576782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.614556+0000","last_change":"2024-09-10T19:01:46.348007+0000","last_active":"2024-09-10T19:01:52.614556+0000","last_peered":"2024-09-10T19:01:52.614556+0000","last_clean":"2024-09-10T19:01:52.614556+0000","last_became_active":"2024-09-10T19:01:46.347710+0000","last_became_peered":"2024-09-10T19:01:46.347710+0000","last_unstale":"2024-09-10T19:01:52.614556+0000","last_undegraded":"2024-09-10T19:01:52.614556+0000","last_fullsized":"2024-09-10T19:01:52.614556+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:51:16.186103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977001+0000","last_change":"2024-09-10T19:01:40.314466+0000","last_active":"2024-09-10T19:01:51.977001+0000","last_peered":"2024-09-10T19:01:51.977001+0000","last_clean":"2024-09-10T19:01:51.977001+0000","last_became_active":"2024-09-10T19:01:40.314031+0000","last_became_peered":"2024-09-10T19:01:40.314031+0000","last_unstale":"2024-09-10T19:01:51.977001+0000","last_undegraded":"2024-09-10T19:01:51.977001+0000","last_fullsized":"2024-09-10T19:01:51.977001+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:08:42.489979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"59'11","reported_seq":35,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.608918+0000","last_change":"2024-09-10T19:01:42.327954+0000","last_active":"2024-09-10T19:01:52.608918+0000","last_peered":"2024-09-10T19:01:52.608918+0000","last_clean":"2024-09-10T19:01:52.608918+0000","last_became_active":"2024-09-10T19:01:42.327652+0000","last_became_peered":"2024-09-10T19:01:42.327652+0000","last_unstale":"2024-09-10T19:01:52.608918+0000","last_undegraded":"2024-09-10T19:01:52.608918+0000","last_fullsized":"2024-09-10T19:01:52.608918+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:13:34.608994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.336458+0000","last_change":"2024-09-10T19:01:44.336159+0000","last_active":"2024-09-10T19:01:51.336458+0000","last_peered":"2024-09-10T19:01:51.336458+0000","last_clean":"2024-09-10T19:01:51.336458+0000","last_became_active":"2024-09-10T19:01:44.335815+0000","last_became_peered":"2024-09-10T19:01:44.335815+0000","last_unstale":"2024-09-10T19:01:51.336458+0000","last_undegraded":"2024-09-10T19:01:51.336458+0000","last_fullsized":"2024-09-10T19:01:51.336458+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:43:13.504025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330553+0000","last_change":"2024-09-10T19:01:46.616324+0000","last_active":"2024-09-10T19:01:51.330553+0000","last_peered":"2024-09-10T19:01:51.330553+0000","last_clean":"2024-09-10T19:01:51.330553+0000","last_became_active":"2024-09-10T19:01:46.616045+0000","last_became_peered":"2024-09-10T19:01:46.616045+0000","last_unstale":"2024-09-10T19:01:51.330553+0000","last_undegraded":"2024-09-10T19:01:51.330553+0000","last_fullsized":"2024-09-10T19:01:51.330553+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:23:10.008384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.607768+0000","last_change":"2024-09-10T19:01:40.313651+0000","last_active":"2024-09-10T19:01:52.607768+0000","last_peered":"2024-09-10T19:01:52.607768+0000","last_clean":"2024-09-10T19:01:52.607768+0000","last_became_active":"2024-09-10T19:01:40.313254+0000","last_became_peered":"2024-09-10T19:01:40.313254+0000","last_unstale":"2024-09-10T19:01:52.607768+0000","last_undegraded":"2024-09-10T19:01:52.607768+0000","last_fullsized":"2024-09-10T19:01:52.607768+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:56:10.551627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"59'15","reported_seq":41,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.335050+0000","last_change":"2024-09-10T19:01:42.329503+0000","last_active":"2024-09-10T19:01:51.335050+0000","last_peered":"2024-09-10T19:01:51.335050+0000","last_clean":"2024-09-10T19:01:51.335050+0000","last_became_active":"2024-09-10T19:01:42.329219+0000","last_became_peered":"2024-09-10T19:01:42.329219+0000","last_unstale":"2024-09-10T19:01:51.335050+0000","last_undegraded":"2024-09-10T19:01:51.335050+0000","last_fullsized":"2024-09-10T19:01:51.335050+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:14:14.960538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:52.613506+0000","last_change":"2024-09-10T19:01:44.345686+0000","last_active":"2024-09-10T19:01:52.613506+0000","last_peered":"2024-09-10T19:01:52.613506+0000","last_clean":"2024-09-10T19:01:52.613506+0000","last_became_active":"2024-09-10T19:01:44.345152+0000","last_became_peered":"2024-09-10T19:01:44.345152+0000","last_unstale":"2024-09-10T19:01:52.613506+0000","last_undegraded":"2024-09-10T19:01:52.613506+0000","last_fullsized":"2024-09-10T19:01:52.613506+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:11:53.443027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977453+0000","last_change":"2024-09-10T19:01:46.614616+0000","last_active":"2024-09-10T19:01:51.977453+0000","last_peered":"2024-09-10T19:01:51.977453+0000","last_clean":"2024-09-10T19:01:51.977453+0000","last_became_active":"2024-09-10T19:01:46.614062+0000","last_became_peered":"2024-09-10T19:01:46.614062+0000","last_unstale":"2024-09-10T19:01:51.977453+0000","last_undegraded":"2024-09-10T19:01:51.977453+0000","last_fullsized":"2024-09-10T19:01:51.977453+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:45.327907+0000","last_clean_scrub_stamp":"2024-09-10T19:01:45.327907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:18:46.911945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.976995+0000","last_change":"2024-09-10T19:01:40.312236+0000","last_active":"2024-09-10T19:01:51.976995+0000","last_peered":"2024-09-10T19:01:51.976995+0000","last_clean":"2024-09-10T19:01:51.976995+0000","last_became_active":"2024-09-10T19:01:40.311960+0000","last_became_peered":"2024-09-10T19:01:40.311960+0000","last_unstale":"2024-09-10T19:01:51.976995+0000","last_undegraded":"2024-09-10T19:01:51.976995+0000","last_fullsized":"2024-09-10T19:01:51.976995+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:39.289402+0000","last_clean_scrub_stamp":"2024-09-10T19:01:39.289402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:22:12.438528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"59'12","reported_seq":39,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.977287+0000","last_change":"2024-09-10T19:01:42.328241+0000","last_active":"2024-09-10T19:01:51.977287+0000","last_peered":"2024-09-10T19:01:51.977287+0000","last_clean":"2024-09-10T19:01:51.977287+0000","last_became_active":"2024-09-10T19:01:42.327964+0000","last_became_peered":"2024-09-10T19:01:42.327964+0000","last_unstale":"2024-09-10T19:01:51.977287+0000","last_undegraded":"2024-09-10T19:01:51.977287+0000","last_fullsized":"2024-09-10T19:01:51.977287+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:41.300426+0000","last_clean_scrub_stamp":"2024-09-10T19:01:41.300426+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:58:37.081806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":14,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-10T19:01:51.330408+0000","last_change":"2024-09-10T19:01:44.349855+0000","last_active":"2024-09-10T19:01:51.330408+0000","last_peered":"2024-09-10T19:01:51.330408+0000","last_clean":"2024-09-10T19:01:51.330408+0000","last_became_active":"2024-09-10T19:01:44.349548+0000","last_became_peered":"2024-09-10T19:01:44.349548+0000","last_unstale":"2024-09-10T19:01:51.330408+0000","last_undegraded":"2024-09-10T19:01:51.330408+0000","last_fullsized":"2024-09-10T19:01:51.330408+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":55,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:01:43.313182+0000","last_clean_scrub_stamp":"2024-09-10T19:01:43.313182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:06:43.238606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":391,"ondisk_log_size":391,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":81,"num_read_kb":76,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430231,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298532,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446620,"statfs":{"total":95995035648,"available":95689338880,"internally_reserved":0,"allocated":1208320,"data_stored":822643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.374},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.45500000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.42899999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.439},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.47899999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.41399999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.28499999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":0.32600000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.46400000000000002}]}]},{"osd":6,"up_from":42,"seq":180388626461,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298512,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446640,"statfs":{"total":95995035648,"available":95689359360,"internally_reserved":0,"allocated":1187840,"data_stored":822042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.499}]},{"osd":1,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.55700000000000005}]},{"osd":2,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.52100000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.51400000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.38},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"last":0.33900000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:02:04 2024","interfaces":[{"interface":"back","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.39200000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:02:24 2024","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.436,"15min":0.436},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.44},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.41499999999999998}]}]},{"osd":5,"up_from":36,"seq":154618822688,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297940,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447212,"statfs":{"total":95995035648,"available":95689945088,"internally_reserved":0,"allocated":598016,"data_stored":231972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48799999999999999,"15min":0.48699999999999999},"min":{"1min":0.40100000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.59699999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.498},"min":{"1min":0.374,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.46899999999999997}]},{"osd":1,"last update":"Tue Sep 10 19:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.53000000000000003,"15min":0.52800000000000002},"min":{"1min":0.42799999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.51900000000000002,"15min":0.52400000000000002},"min":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.442}]},{"osd":2,"last update":"Tue Sep 10 19:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.34499999999999997,"5min":0.31,"15min":0.31},"max":{"1min":0.61599999999999999,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.434},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.40899999999999997}]},{"osd":3,"last update":"Tue Sep 10 19:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.48999999999999999,"15min":0.48799999999999999},"min":{"1min":0.36499999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.63500000000000001,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.42499999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:02:39 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.36199999999999999,"15min":0.35599999999999998},"min":{"1min":0.27700000000000002,"5min":0.221,"15min":0.221},"max":{"1min":0.53000000000000003,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.433,"5min":0.377,"15min":0.36799999999999999},"min":{"1min":0.29199999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.37}]},{"osd":6,"last update":"Tue Sep 10 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.39900000000000002,"15min":0.39200000000000002},"min":{"1min":0.32500000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.40799999999999997,"15min":0.40000000000000002},"min":{"1min":0.32300000000000001,"5min":0.246,"15min":0.246},"max":{"1min":0.64900000000000002,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.38500000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.30399999999999999}]}]},{"osd":4,"up_from":30,"seq":128849018917,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297952,"kb_used_data":596,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447200,"statfs":{"total":95995035648,"available":95689932800,"internally_reserved":0,"allocated":610304,"data_stored":231798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.47499999999999998,"15min":0.47299999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.50900000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46700000000000003,"15min":0.46400000000000002},"min":{"1min":0.35999999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.39800000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.71199999999999997,"5min":0.754,"15min":0.754},"last":0.57799999999999996}]},{"osd":2,"last update":"Tue Sep 10 19:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71199999999999997,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.499,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.65300000000000002,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.65200000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.502,"15min":0.501},"min":{"1min":0.35999999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.70299999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.49199999999999999,"15min":0.48799999999999999},"min":{"1min":0.371,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.622}]},{"osd":5,"last update":"Tue Sep 10 19:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.38500000000000001,"15min":0.377},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.29399999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.54000000000000004,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.42799999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:03:05 2024","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.40799999999999997,"15min":0.40200000000000002},"min":{"1min":0.31900000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.629,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.41799999999999998,"15min":0.41399999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.65500000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.48799999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.47599999999999998}]}]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297980,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447172,"statfs":{"total":95995035648,"available":95689904128,"internally_reserved":0,"allocated":638976,"data_stored":232839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.36699999999999999,"15min":0.374},"min":{"1min":0.158,"5min":0.158,"15min":0.158},"max":{"1min":0.503,"5min":0.50600000000000001,"15min":0.50600000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.39600000000000002,"15min":0.40799999999999997},"min":{"1min":0.14599999999999999,"5min":0.14599999999999999,"15min":0.14599999999999999},"max":{"1min":0.5,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.26000000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.40100000000000002,"15min":0.42099999999999999},"min":{"1min":0.20000000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.48899999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.41099999999999998,"15min":0.42099999999999999},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.56100000000000005,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.32300000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:02:56 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42299999999999999,"15min":0.437},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.60199999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.39500000000000002,"15min":0.40699999999999997},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.497,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.33400000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:02:17 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.66100000000000003,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.60399999999999998,"15min":0.61199999999999999},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.71099999999999997,"5min":0.878,"15min":0.878},"last":0.432}]},{"osd":5,"last update":"Tue Sep 10 19:02:41 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.52700000000000002,"15min":0.53100000000000003},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.37}]},{"osd":6,"last update":"Tue Sep 10 19:03:04 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55000000000000004,"15min":0.54800000000000004},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.39400000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:02:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.42099999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297940,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447212,"statfs":{"total":95995035648,"available":95689945088,"internally_reserved":0,"allocated":598016,"data_stored":232977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:34 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.34699999999999998,"15min":0.35199999999999998},"min":{"1min":0.20200000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.56399999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.376},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.37,"15min":0.36099999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.44900000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:02:34 2024","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.38100000000000001,"15min":0.38300000000000001},"min":{"1min":0.25800000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.54200000000000004,"5min":0.58599999999999997,"15min":0.58599999999999997},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.40699999999999997,"15min":0.40899999999999997},"min":{"1min":0.28799999999999998,"5min":0.216,"15min":0.216},"max":{"1min":0.53300000000000003,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.39900000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.442,"15min":0.46000000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.61399999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.39600000000000002,"15min":0.41099999999999998},"min":{"1min":0.25,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.48099999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35599999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:02:14 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.52100000000000002,"15min":0.52900000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.54400000000000004,"15min":0.55000000000000004},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.50600000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:02:42 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52400000000000002,"15min":0.52300000000000002},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.72499999999999998,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.497,"15min":0.495},"min":{"1min":0.34799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.65700000000000003,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.438}]},{"osd":6,"last update":"Tue Sep 10 19:03:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.68899999999999995,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.52200000000000002,"15min":0.52300000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.69699999999999995,"5min":0.746,"15min":0.746},"last":0.41399999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:02:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.46400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":297992,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447160,"statfs":{"total":95995035648,"available":95689891840,"internally_reserved":0,"allocated":651264,"data_stored":234148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:02:09 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34300000000000003,"15min":0.34100000000000003},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.49099999999999999,"5min":0.58099999999999996,"15min":0.58099999999999996},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35099999999999998,"15min":0.34699999999999998},"min":{"1min":0.23200000000000001,"5min":0.223,"15min":0.223},"max":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"last":0.29699999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:02:35 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.371,"15min":0.36499999999999999},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.55400000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40300000000000002,"15min":0.40100000000000002},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.32200000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:03:00 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.41099999999999998,"15min":0.42199999999999999},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.42799999999999999,"15min":0.437},"min":{"1min":0.28000000000000003,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.34200000000000003}]},{"osd":4,"last update":"Tue Sep 10 19:02:14 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.55300000000000005,"15min":0.55000000000000004},"min":{"1min":0.443,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.55700000000000005}]},{"osd":5,"last update":"Tue Sep 10 19:02:43 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51100000000000001,"15min":0.51200000000000001},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.54800000000000004,"15min":0.54400000000000004},"min":{"1min":0.45100000000000001,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.69899999999999995,"5min":0.751,"15min":0.751},"last":0.52600000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:03:03 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.745,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.54300000000000004,"15min":0.54800000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.62,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.56999999999999995}]},{"osd":7,"last update":"Tue Sep 10 19:02:27 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.58699999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298540,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446612,"statfs":{"total":95995035648,"available":95689330688,"internally_reserved":0,"allocated":1216512,"data_stored":823576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:02:10 2024","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.33200000000000002,"15min":0.312},"min":{"1min":0.23699999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.499,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.371,"5min":0.33700000000000002,"15min":0.317},"min":{"1min":0.245,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.52200000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"last":0.36399999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:02:37 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.38900000000000001,"15min":0.371},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.434},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.38300000000000001,"15min":0.36699999999999999},"min":{"1min":0.24299999999999999,"5min":0.221,"15min":0.221},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.32700000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:02:57 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.45200000000000001,"15min":0.45000000000000001},"min":{"1min":0.29999999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.44,"15min":0.44},"min":{"1min":0.31900000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.4970000000000001,"5min":1.4970000000000001,"15min":1.4970000000000001},"last":0.38}]},{"osd":4,"last update":"Tue Sep 10 19:02:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.54300000000000004,"15min":0.53800000000000003},"min":{"1min":0.374,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.57199999999999995,"15min":0.56599999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.64200000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:02:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.61099999999999999,"15min":0.61599999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.76600000000000001,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.74399999999999999,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.60799999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:03:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.76400000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60799999999999998,"15min":0.60899999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.72199999999999998,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.65900000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:02:26 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.67300000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:03:12.584 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-10T19:03:12.584 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-10T19:03:12.584 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-10T19:03:12.584 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 -- ceph health --format=json 2024-09-10T19:03:12.832 INFO:teuthology.orchestra.run.smithi119.stderr:Inferring config /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:03:13.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:13 smithi194 ceph-mon[32464]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:03:13.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:13 smithi119 ceph-mon[29577]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:03:13.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:13 smithi119 ceph-mon[40674]: from='client.14691 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:03:14.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:14 smithi119 ceph-mon[29577]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:14.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:14 smithi119 ceph-mon[40674]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:14.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:14 smithi194 ceph-mon[32464]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:14.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.572+0000 7fc43c06e640 1 -- 172.21.15.119:0/3913120375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4340fd400 msgr2=0x7fc434109690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:14.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.572+0000 7fc43c06e640 1 --2- 172.21.15.119:0/3913120375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4340fd400 0x7fc434109690 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fc42c009d50 tx=0x7fc42c004c50 comp rx=0 tx=0).stop 2024-09-10T19:03:14.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.572+0000 7fc43c06e640 1 -- 172.21.15.119:0/3913120375 shutdown_connections 2024-09-10T19:03:14.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.572+0000 7fc43c06e640 1 --2- 172.21.15.119:0/3913120375 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4340fd400 0x7fc434109690 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:14.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.572+0000 7fc43c06e640 1 --2- 172.21.15.119:0/3913120375 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4340fca50 0x7fc4340fcec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:14.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.572+0000 7fc43c06e640 1 --2- 172.21.15.119:0/3913120375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4341028e0 0x7fc434102cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:14.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.572+0000 7fc43c06e640 1 -- 172.21.15.119:0/3913120375 >> 172.21.15.119:0/3913120375 conn(0x7fc434073600 msgr2=0x7fc434073a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:14.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.572+0000 7fc43c06e640 1 -- 172.21.15.119:0/3913120375 shutdown_connections 2024-09-10T19:03:14.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.572+0000 7fc43c06e640 1 -- 172.21.15.119:0/3913120375 wait complete. 2024-09-10T19:03:14.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc43c06e640 1 Processor -- start 2024-09-10T19:03:14.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc43c06e640 1 -- start start 2024-09-10T19:03:14.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc43c06e640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4340fca50 0x7fc434112bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:14.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc43c06e640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4340fd400 0x7fc434113100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:14.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc43c06e640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4341028e0 0x7fc43410cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:14.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc43c06e640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43410d3b0 con 0x7fc4341028e0 2024-09-10T19:03:14.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc43c06e640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43410d520 con 0x7fc4340fd400 2024-09-10T19:03:14.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc43c06e640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43410d6e0 con 0x7fc4340fca50 2024-09-10T19:03:14.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc439e0a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4340fca50 0x7fc434112bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:14.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc43a60b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4341028e0 0x7fc43410cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:14.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc439e0a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4340fca50 0x7fc434112bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:42328/0 (socket says 172.21.15.119:42328) 2024-09-10T19:03:14.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc439e0a640 1 -- 172.21.15.119:0/2590278272 learned_addr learned my addr 172.21.15.119:0/2590278272 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:03:14.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc439609640 1 --2- 172.21.15.119:0/2590278272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4340fd400 0x7fc434113100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:14.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc439e0a640 1 -- 172.21.15.119:0/2590278272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4340fd400 msgr2=0x7fc434113100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:14.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc439e0a640 1 --2- 172.21.15.119:0/2590278272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4340fd400 0x7fc434113100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:14.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc439e0a640 1 -- 172.21.15.119:0/2590278272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4341028e0 msgr2=0x7fc43410cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:14.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc439e0a640 1 --2- 172.21.15.119:0/2590278272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4341028e0 0x7fc43410cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:14.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc439e0a640 1 -- 172.21.15.119:0/2590278272 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc430005040 con 0x7fc4340fca50 2024-09-10T19:03:14.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc43a60b640 1 --2- 172.21.15.119:0/2590278272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4341028e0 0x7fc43410cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:03:14.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.573+0000 7fc439e0a640 1 --2- 172.21.15.119:0/2590278272 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4340fca50 0x7fc434112bc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc42400b810 tx=0x7fc4240091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:03:14.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.574+0000 7fc42a7fc640 1 -- 172.21.15.119:0/2590278272 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42400be60 con 0x7fc4340fca50 2024-09-10T19:03:14.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.574+0000 7fc42a7fc640 1 -- 172.21.15.119:0/2590278272 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fc42400cba0 con 0x7fc4340fca50 2024-09-10T19:03:14.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.574+0000 7fc43c06e640 1 -- 172.21.15.119:0/2590278272 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc42c0098b0 con 0x7fc4340fca50 2024-09-10T19:03:14.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.574+0000 7fc42a7fc640 1 -- 172.21.15.119:0/2590278272 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc424015b60 con 0x7fc4340fca50 2024-09-10T19:03:14.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.574+0000 7fc43c06e640 1 -- 172.21.15.119:0/2590278272 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc434119a50 con 0x7fc4340fca50 2024-09-10T19:03:14.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.575+0000 7fc43c06e640 1 -- 172.21.15.119:0/2590278272 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4340fe000 con 0x7fc4340fca50 2024-09-10T19:03:14.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.575+0000 7fc42a7fc640 1 -- 172.21.15.119:0/2590278272 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7fc424012070 con 0x7fc4340fca50 2024-09-10T19:03:14.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.576+0000 7fc42a7fc640 1 --2- 172.21.15.119:0/2590278272 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc42006eed0 0x7fc420071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:03:14.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.576+0000 7fc439609640 1 --2- 172.21.15.119:0/2590278272 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc42006eed0 0x7fc420071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:03:14.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.576+0000 7fc42a7fc640 1 -- 172.21.15.119:0/2590278272 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6168+0+0 (secure 0 0 0) 0x7fc42408db10 con 0x7fc4340fca50 2024-09-10T19:03:14.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.576+0000 7fc439609640 1 --2- 172.21.15.119:0/2590278272 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc42006eed0 0x7fc420071380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc430007fd0 tx=0x7fc43000b040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:03:14.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.580+0000 7fc42a7fc640 1 -- 172.21.15.119:0/2590278272 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fc42405aaa0 con 0x7fc4340fca50 2024-09-10T19:03:14.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.767+0000 7fc43c06e640 1 -- 172.21.15.119:0/2590278272 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc43410bdc0 con 0x7fc4340fca50 2024-09-10T19:03:14.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.767+0000 7fc42a7fc640 1 -- 172.21.15.119:0/2590278272 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fc42405a440 con 0x7fc4340fca50 2024-09-10T19:03:14.769 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-10T19:03:14.769 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-10T19:03:14.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.769+0000 7fc43c06e640 1 -- 172.21.15.119:0/2590278272 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc42006eed0 msgr2=0x7fc420071380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:14.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.769+0000 7fc43c06e640 1 --2- 172.21.15.119:0/2590278272 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc42006eed0 0x7fc420071380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc430007fd0 tx=0x7fc43000b040 comp rx=0 tx=0).stop 2024-09-10T19:03:14.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.769+0000 7fc43c06e640 1 -- 172.21.15.119:0/2590278272 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4340fca50 msgr2=0x7fc434112bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:03:14.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.769+0000 7fc43c06e640 1 --2- 172.21.15.119:0/2590278272 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4340fca50 0x7fc434112bc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc42400b810 tx=0x7fc4240091d0 comp rx=0 tx=0).stop 2024-09-10T19:03:14.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.769+0000 7fc43c06e640 1 -- 172.21.15.119:0/2590278272 shutdown_connections 2024-09-10T19:03:14.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.769+0000 7fc43c06e640 1 --2- 172.21.15.119:0/2590278272 >> v2:172.21.15.119:6800/3118961733 conn(0x7fc42006eed0 0x7fc420071380 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:14.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.769+0000 7fc43c06e640 1 --2- 172.21.15.119:0/2590278272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4341028e0 0x7fc43410cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:14.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.769+0000 7fc43c06e640 1 --2- 172.21.15.119:0/2590278272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4340fd400 0x7fc434113100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:14.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.769+0000 7fc43c06e640 1 --2- 172.21.15.119:0/2590278272 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4340fca50 0x7fc434112bc0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:03:14.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.769+0000 7fc43c06e640 1 -- 172.21.15.119:0/2590278272 >> 172.21.15.119:0/2590278272 conn(0x7fc434073600 msgr2=0x7fc4340fb480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:03:14.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.769+0000 7fc43c06e640 1 -- 172.21.15.119:0/2590278272 shutdown_connections 2024-09-10T19:03:14.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-10T19:03:14.769+0000 7fc43c06e640 1 -- 172.21.15.119:0/2590278272 wait complete. 2024-09-10T19:03:15.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2590278272' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-10T19:03:15.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2590278272' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-10T19:03:15.371 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-10T19:03:15.371 INFO:tasks.cephadm:Setup complete, yielding 2024-09-10T19:03:15.372 INFO:teuthology.run_tasks:Running task workunit... 2024-09-10T19:03:15.388 INFO:tasks.workunit:Pulling workunits from ref fad5356a94e6044ce1062d748daa5a0f2ed53a7d 2024-09-10T19:03:15.389 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-10T19:03:15.389 DEBUG:teuthology.orchestra.run.smithi119:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-10T19:03:15.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:03:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-10T19:03:15.406 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-10T19:03:15.462 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-10T19:03:15.462 DEBUG:teuthology.orchestra.run.smithi119:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-10T19:03:15.517 INFO:tasks.workunit:timeout=3h 2024-09-10T19:03:15.518 INFO:tasks.workunit:cleanup=True 2024-09-10T19:03:15.518 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/adk3798/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout fad5356a94e6044ce1062d748daa5a0f2ed53a7d 2024-09-10T19:03:15.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2590278272' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-10T19:03:15.574 INFO:tasks.workunit.client.0.smithi119.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-10T19:03:16.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-mon[32464]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:16.519 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 bash[66065]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-10T19:03:16.519 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 bash[66065]: Writing manifest to image destination 2024-09-10T19:03:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:16 smithi119 ceph-mon[29577]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:16.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:16 smithi119 ceph-mon[40674]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:16.855 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 podman[66065]: 2024-09-10 19:03:16.479609348 +0000 UTC m=+7.461206434 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-10T19:03:16.855 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 podman[66065]: 2024-09-10 19:03:16.586267393 +0000 UTC m=+7.567864479 container create 87647be4cef28eaca4a224f5fa911d43dbf3e90e7a4e03b8c3a2298f9ba1db9f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T19:03:16.855 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 podman[66065]: 2024-09-10 19:03:16.85487136 +0000 UTC m=+7.836468450 container init 87647be4cef28eaca4a224f5fa911d43dbf3e90e7a4e03b8c3a2298f9ba1db9f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T19:03:17.268 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 podman[66065]: 2024-09-10 19:03:16.860279333 +0000 UTC m=+7.841876419 container start 87647be4cef28eaca4a224f5fa911d43dbf3e90e7a4e03b8c3a2298f9ba1db9f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T19:03:17.269 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.864Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-10T19:03:17.269 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.864Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-10T19:03:17.269 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.864Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-10T19:03:17.269 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.864Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-10T19:03:17.269 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.864Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-10T19:03:17.269 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-10T19:03:17.269 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-10T19:03:17.269 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=arp 2024-09-10T19:03:17.269 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-10T19:03:17.269 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-10T19:03:17.269 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-10T19:03:17.269 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-10T19:03:17.270 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-10T19:03:17.270 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-10T19:03:17.270 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-10T19:03:17.270 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-10T19:03:17.270 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=edac 2024-09-10T19:03:17.270 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-10T19:03:17.270 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-10T19:03:17.270 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-10T19:03:17.271 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-10T19:03:17.271 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-10T19:03:17.271 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-10T19:03:17.271 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-10T19:03:17.271 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-10T19:03:17.271 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-10T19:03:17.271 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-10T19:03:17.271 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-10T19:03:17.271 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-10T19:03:17.271 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-10T19:03:17.271 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-10T19:03:17.271 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-10T19:03:17.271 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-10T19:03:17.271 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=os 2024-09-10T19:03:17.272 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-10T19:03:17.272 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-10T19:03:17.272 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-10T19:03:17.272 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-10T19:03:17.272 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-10T19:03:17.272 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-10T19:03:17.272 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-10T19:03:17.272 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=stat 2024-09-10T19:03:17.272 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-10T19:03:17.272 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-10T19:03:17.272 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-10T19:03:17.272 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=time 2024-09-10T19:03:17.272 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-10T19:03:17.272 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=uname 2024-09-10T19:03:17.273 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-10T19:03:17.273 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-10T19:03:17.273 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.865Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-10T19:03:17.273 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.866Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-10T19:03:17.273 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b[66141]: ts=2024-09-10T19:03:16.866Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-10T19:03:17.273 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 bash[66065]: 87647be4cef28eaca4a224f5fa911d43dbf3e90e7a4e03b8c3a2298f9ba1db9f 2024-09-10T19:03:17.273 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:03:16 smithi194 systemd[1]: Started Ceph node-exporter.b for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:03:18.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:03:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:03:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-09-10T19:03:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-09-10T19:03:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-10T19:03:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T19:03:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:17 smithi194 ceph-mon[32464]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:18.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:18.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:03:18.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.280 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.280 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.280 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.281 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.281 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.281 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-09-10T19:03:18.281 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-09-10T19:03:18.281 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-10T19:03:18.281 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T19:03:18.281 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[29577]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:18.281 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:18.281 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:03:18.281 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.281 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:18.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]: dispatch 2024-09-10T19:03:18.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-09-10T19:03:18.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-09-10T19:03:18.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T19:03:18.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[40674]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:18.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:18.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:03:18.282 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:19 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]': finished 2024-09-10T19:03:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:19 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-10T19:03:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:19 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-09-10T19:03:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:19 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-10T19:03:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:19 smithi119 ceph-mon[29577]: osdmap e62: 8 total, 8 up, 8 in 2024-09-10T19:03:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:19 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:19 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]': finished 2024-09-10T19:03:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:19 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-10T19:03:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:19 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-09-10T19:03:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:19 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-10T19:03:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:19 smithi119 ceph-mon[40674]: osdmap e62: 8 total, 8 up, 8 in 2024-09-10T19:03:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:19 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:19.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:19 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.8", "id": [7, 2]}]': finished 2024-09-10T19:03:19.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:19 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-10T19:03:19.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:19 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-09-10T19:03:19.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:19 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-10T19:03:19.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:19 smithi194 ceph-mon[32464]: osdmap e62: 8 total, 8 up, 8 in 2024-09-10T19:03:19.398 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:19 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:20.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:20 smithi119 ceph-mon[29577]: osdmap e63: 8 total, 8 up, 8 in 2024-09-10T19:03:20.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:20 smithi119 ceph-mon[29577]: pgmap v36: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:20.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:20 smithi119 ceph-mon[40674]: osdmap e63: 8 total, 8 up, 8 in 2024-09-10T19:03:20.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:20 smithi119 ceph-mon[40674]: pgmap v36: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:20.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:20 smithi194 ceph-mon[32464]: osdmap e63: 8 total, 8 up, 8 in 2024-09-10T19:03:20.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:20 smithi194 ceph-mon[32464]: pgmap v36: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:20.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:03:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:03:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:03:21.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:21 smithi194 ceph-mon[32464]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-10T19:03:21.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:21 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:21.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:21 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:21.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:21 smithi119 ceph-mon[29577]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-10T19:03:21.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:21 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:21.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:21 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:21.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:21 smithi119 ceph-mon[40674]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-10T19:03:21.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:21 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:21.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:21 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:22.430 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:22 smithi119 ceph-mon[40674]: pgmap v37: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:22.430 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:22.430 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:22.430 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:03:22.430 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:03:22.430 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:22 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:22.430 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:22 smithi119 ceph-mon[29577]: pgmap v37: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:22.430 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:22.430 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:22.431 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:03:22.431 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:03:22.431 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:22 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:22.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:22 smithi194 ceph-mon[32464]: pgmap v37: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:03:22.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:22.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:03:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:03:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:22 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:23.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:23 smithi119 ceph-mon[29577]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-10T19:03:23.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:23 smithi119 ceph-mon[29577]: Reconfiguring daemon alertmanager.a on smithi119 2024-09-10T19:03:23.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:23 smithi119 ceph-mon[40674]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-10T19:03:23.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:23 smithi119 ceph-mon[40674]: Reconfiguring daemon alertmanager.a on smithi119 2024-09-10T19:03:23.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:23 smithi194 ceph-mon[32464]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-10T19:03:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:23 smithi194 ceph-mon[32464]: Reconfiguring daemon alertmanager.a on smithi119 2024-09-10T19:03:24.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:24 smithi119 ceph-mon[29577]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2024-09-10T19:03:24.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:24 smithi119 ceph-mon[29577]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-10T19:03:24.513 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:24 smithi119 ceph-mon[29577]: Cluster is now healthy 2024-09-10T19:03:24.513 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:24 smithi119 ceph-mon[40674]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2024-09-10T19:03:24.513 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:24 smithi119 ceph-mon[40674]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-10T19:03:24.513 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:24 smithi119 ceph-mon[40674]: Cluster is now healthy 2024-09-10T19:03:24.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:24 smithi194 ceph-mon[32464]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2024-09-10T19:03:24.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:24 smithi194 ceph-mon[32464]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-10T19:03:24.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:24 smithi194 ceph-mon[32464]: Cluster is now healthy 2024-09-10T19:03:25.566 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:25 smithi119 systemd[1]: Stopping Ceph alertmanager.a for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:03:25.981 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:25 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[75363]: ts=2024-09-10T19:03:25.655Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-10T19:03:25.981 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:25 smithi119 podman[78049]: 2024-09-10 19:03:25.676180429 +0000 UTC m=+0.145546079 container died 7e27161edb5ac44951c597d1d3dcfa55918121ee65e7b899879936c3881457b1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:26.231 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:26 smithi119 ceph-mon[29577]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2024-09-10T19:03:26.231 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:26 smithi119 podman[78049]: 2024-09-10 19:03:26.078869146 +0000 UTC m=+0.548234789 container cleanup 7e27161edb5ac44951c597d1d3dcfa55918121ee65e7b899879936c3881457b1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:26.231 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:26 smithi119 bash[78049]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a 2024-09-10T19:03:26.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:26 smithi194 ceph-mon[32464]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2024-09-10T19:03:26.556 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:26 smithi119 ceph-mon[40674]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2024-09-10T19:03:26.557 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:26 smithi119 podman[78068]: 2024-09-10 19:03:26.455585803 +0000 UTC m=+0.780825225 container remove 7e27161edb5ac44951c597d1d3dcfa55918121ee65e7b899879936c3881457b1 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:26.557 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:26 smithi119 podman[78068]: 2024-09-10 19:03:26.556381979 +0000 UTC m=+0.881621409 volume remove fe8babc2ce04fdc80c79235e44755d2e33bc4dbe12d8661ba6d9b2ccab824db8 2024-09-10T19:03:26.816 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:26 smithi119 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-10T19:03:26.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:26 smithi119 systemd[1]: Stopped Ceph alertmanager.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:03:26.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:26 smithi119 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@alertmanager.a.service: Consumed 1.002s CPU time. 2024-09-10T19:03:26.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:26 smithi119 systemd[1]: Starting Ceph alertmanager.a for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:03:27.296 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:26 smithi119 podman[78160]: 2024-09-10 19:03:26.816795477 +0000 UTC m=+0.016570740 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-10T19:03:27.296 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:26 smithi119 podman[78160]: 2024-09-10 19:03:26.923929114 +0000 UTC m=+0.123704378 volume create 71aecd5e36d7b9b3839a7b39bb8e3ba462f62410db9199a8f8e59fcd33c34a47 2024-09-10T19:03:27.296 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:27 smithi119 podman[78160]: 2024-09-10 19:03:27.041050826 +0000 UTC m=+0.240826089 container create 1b10ef1281739cdf6e840b404c3b9b655877e92e0e3816b79b9e01e11e39f35f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:27.816 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:27 smithi119 podman[78160]: 2024-09-10 19:03:27.436093913 +0000 UTC m=+0.635869178 container init 1b10ef1281739cdf6e840b404c3b9b655877e92e0e3816b79b9e01e11e39f35f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:27.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:27 smithi119 podman[78160]: 2024-09-10 19:03:27.441816788 +0000 UTC m=+0.641592051 container start 1b10ef1281739cdf6e840b404c3b9b655877e92e0e3816b79b9e01e11e39f35f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:27.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:27 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:03:27.453Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-10T19:03:27.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:27 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:03:27.454Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-10T19:03:27.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:27 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:03:27.455Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.119 port=9094 2024-09-10T19:03:27.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:27 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:03:27.456Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-10T19:03:27.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:27 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:03:27.484Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-10T19:03:27.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:27 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:03:27.484Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-10T19:03:27.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:27 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:03:27.487Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-10T19:03:27.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:27 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:03:27.487Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-10T19:03:27.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:27 smithi119 bash[78160]: 1b10ef1281739cdf6e840b404c3b9b655877e92e0e3816b79b9e01e11e39f35f 2024-09-10T19:03:27.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:27 smithi119 systemd[1]: Started Ceph alertmanager.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:03:28.231 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:03:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:03:28.516 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:28 smithi194 ceph-mon[32464]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 24 B/s, 2 objects/s recovering 2024-09-10T19:03:28.516 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:28 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:28.516 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:28 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:28.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:28 smithi119 ceph-mon[29577]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 24 B/s, 2 objects/s recovering 2024-09-10T19:03:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:28 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:28 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:28.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:28 smithi119 ceph-mon[40674]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 24 B/s, 2 objects/s recovering 2024-09-10T19:03:28.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:28 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:28.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:28 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:29.456 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:29 smithi194 ceph-mon[32464]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-10T19:03:29.457 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:29 smithi194 ceph-mon[32464]: Reconfiguring daemon prometheus.a on smithi194 2024-09-10T19:03:29.457 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:29 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:29.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:29 smithi119 ceph-mon[29577]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-10T19:03:29.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:29 smithi119 ceph-mon[29577]: Reconfiguring daemon prometheus.a on smithi194 2024-09-10T19:03:29.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:29 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:29.567 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:29 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:03:29.457Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000573918s 2024-09-10T19:03:29.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:29 smithi119 ceph-mon[40674]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-10T19:03:29.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:29 smithi119 ceph-mon[40674]: Reconfiguring daemon prometheus.a on smithi194 2024-09-10T19:03:29.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:29 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:30.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:30 smithi194 ceph-mon[32464]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s; 21 B/s, 1 objects/s recovering 2024-09-10T19:03:30.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:03:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:03:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:03:30.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:30 smithi119 ceph-mon[29577]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s; 21 B/s, 1 objects/s recovering 2024-09-10T19:03:30.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:30 smithi119 ceph-mon[40674]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s; 21 B/s, 1 objects/s recovering 2024-09-10T19:03:30.935 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:30 smithi194 systemd[1]: Stopping Ceph prometheus.a for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:03:31.235 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[63818]: ts=2024-09-10T19:03:30.934Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-10T19:03:31.235 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[63818]: ts=2024-09-10T19:03:30.934Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-10T19:03:31.235 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[63818]: ts=2024-09-10T19:03:30.934Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-10T19:03:31.235 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[63818]: ts=2024-09-10T19:03:30.934Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-10T19:03:31.235 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[63818]: ts=2024-09-10T19:03:30.934Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-10T19:03:31.235 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[63818]: ts=2024-09-10T19:03:30.934Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-10T19:03:31.235 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[63818]: ts=2024-09-10T19:03:30.934Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-10T19:03:31.235 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[63818]: ts=2024-09-10T19:03:30.934Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-10T19:03:31.235 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[63818]: ts=2024-09-10T19:03:30.934Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-10T19:03:31.235 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[63818]: ts=2024-09-10T19:03:30.935Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-10T19:03:31.236 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[63818]: ts=2024-09-10T19:03:30.935Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-10T19:03:31.236 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[63818]: ts=2024-09-10T19:03:30.935Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-10T19:03:31.236 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:30 smithi194 podman[67178]: 2024-09-10 19:03:30.944913495 +0000 UTC m=+0.102556471 container died 4eed504aa070787c70bca0b8c95d542242feb43f9ddebcad9d52a974a626b1a7 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:31.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:31 smithi194 podman[67178]: 2024-09-10 19:03:31.323995822 +0000 UTC m=+0.481638795 container cleanup 4eed504aa070787c70bca0b8c95d542242feb43f9ddebcad9d52a974a626b1a7 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:31.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:31 smithi194 bash[67178]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a 2024-09-10T19:03:31.955 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:31 smithi194 podman[67196]: 2024-09-10 19:03:31.600624763 +0000 UTC m=+0.654086655 container remove 4eed504aa070787c70bca0b8c95d542242feb43f9ddebcad9d52a974a626b1a7 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:31.955 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:31 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-10T19:03:31.955 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:31 smithi194 systemd[1]: Stopped Ceph prometheus.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:03:31.955 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:31 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@prometheus.a.service: Consumed 1.104s CPU time. 2024-09-10T19:03:31.955 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:31 smithi194 systemd[1]: Starting Ceph prometheus.a for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:03:32.208 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:31 smithi194 podman[67294]: 2024-09-10 19:03:31.857774187 +0000 UTC m=+0.016527400 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-10T19:03:32.209 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:31 smithi194 podman[67294]: 2024-09-10 19:03:31.977320404 +0000 UTC m=+0.136073610 container create ea606254366cec508b61d18b29d322816fc1e209da1e6cc57c7b8e4ed18af53a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:32.209 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 podman[67294]: 2024-09-10 19:03:32.203469737 +0000 UTC m=+0.362222949 container init ea606254366cec508b61d18b29d322816fc1e209da1e6cc57c7b8e4ed18af53a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:32.209 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 podman[67294]: 2024-09-10 19:03:32.208075578 +0000 UTC m=+0.366828784 container start ea606254366cec508b61d18b29d322816fc1e209da1e6cc57c7b8e4ed18af53a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:03:32.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-mon[32464]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 18 B/s, 1 objects/s recovering 2024-09-10T19:03:32.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:32.519 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.232Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-10T19:03:32.519 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.232Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-10T19:03:32.519 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.232Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi194 (none))" 2024-09-10T19:03:32.519 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.232Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-10T19:03:32.519 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.232Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-10T19:03:32.519 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.234Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-10T19:03:32.519 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.235Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-10T19:03:32.519 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.235Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-10T19:03:32.519 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.235Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-10T19:03:32.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.238Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-10T19:03:32.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.238Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.677µs 2024-09-10T19:03:32.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.238Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-10T19:03:32.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.238Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-10T19:03:32.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.238Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-10T19:03:32.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.238Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.269µs wal_replay_duration=524.806µs wbl_replay_duration=125ns total_replay_duration=561.099µs 2024-09-10T19:03:32.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.239Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-10T19:03:32.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.239Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-10T19:03:32.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.239Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-10T19:03:32.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.255Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.694655ms db_storage=725ns remote_storage=1.363µs web_handler=373ns query_engine=546ns scrape=497.369µs scrape_sd=32.23µs notify=44.644µs notify_sd=17.659µs rules=14.873578ms tracing=6.222µs 2024-09-10T19:03:32.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.255Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-10T19:03:32.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:03:32.255Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-10T19:03:32.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 bash[67294]: ea606254366cec508b61d18b29d322816fc1e209da1e6cc57c7b8e4ed18af53a 2024-09-10T19:03:32.520 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:03:32 smithi194 systemd[1]: Started Ceph prometheus.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:03:32.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:32 smithi119 ceph-mon[29577]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 18 B/s, 1 objects/s recovering 2024-09-10T19:03:32.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:32.567 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:32] ENGINE Bus STOPPING 2024-09-10T19:03:32.567 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T19:03:32.567 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:32] ENGINE Bus STOPPED 2024-09-10T19:03:32.567 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:32] ENGINE Bus STARTING 2024-09-10T19:03:32.567 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:32] ENGINE Serving on http://:::9283 2024-09-10T19:03:32.567 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:32] ENGINE Bus STARTED 2024-09-10T19:03:32.567 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:32] ENGINE Bus STOPPING 2024-09-10T19:03:32.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:32 smithi119 ceph-mon[40674]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2 op/s; 18 B/s, 1 objects/s recovering 2024-09-10T19:03:32.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:33.311 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:32] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T19:03:33.311 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:32] ENGINE Bus STOPPED 2024-09-10T19:03:33.311 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:32] ENGINE Bus STARTING 2024-09-10T19:03:33.311 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:33] ENGINE Serving on http://:::9283 2024-09-10T19:03:33.311 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:33] ENGINE Bus STARTED 2024-09-10T19:03:33.311 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:33] ENGINE Bus STOPPING 2024-09-10T19:03:33.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T19:03:33.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T19:03:33.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi119.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T19:03:33.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi119.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T19:03:33.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T19:03:33.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T19:03:33.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi194.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T19:03:33.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi194.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T19:03:33.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T19:03:33.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T19:03:33.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi194.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T19:03:33.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi194.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T19:03:33.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:03:33.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T19:03:33.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T19:03:33.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi119.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T19:03:33.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi119.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T19:03:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T19:03:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T19:03:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi194.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T19:03:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi194.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T19:03:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T19:03:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T19:03:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi194.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T19:03:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi194.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T19:03:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:03:33.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T19:03:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T19:03:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi119.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T19:03:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi119.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T19:03:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T19:03:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T19:03:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi194.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T19:03:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi194.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T19:03:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T19:03:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T19:03:33.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi194.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T19:03:33.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi194.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T19:03:33.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:33.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:03:33.883 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:33] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T19:03:33.883 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:33] ENGINE Bus STOPPED 2024-09-10T19:03:33.883 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:33] ENGINE Bus STARTING 2024-09-10T19:03:33.883 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:33] ENGINE Serving on http://:::9283 2024-09-10T19:03:33.883 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:03:33] ENGINE Bus STARTED 2024-09-10T19:03:34.713 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:34 smithi194 ceph-mon[32464]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 3 op/s; 18 B/s, 1 objects/s recovering 2024-09-10T19:03:34.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:34 smithi119 ceph-mon[29577]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 3 op/s; 18 B/s, 1 objects/s recovering 2024-09-10T19:03:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:34 smithi119 ceph-mon[40674]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 3 op/s; 18 B/s, 1 objects/s recovering 2024-09-10T19:03:36.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:36 smithi194 ceph-mon[32464]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:36.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:36 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:36.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:36 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:36.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:36 smithi119 ceph-mon[29577]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:36.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:36 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:36 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:36 smithi119 ceph-mon[40674]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:36 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:36 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:37.662 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:03:37.460Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003470568s 2024-09-10T19:03:38.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:37 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:38.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:37 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:38.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:37 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:03:38.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:37 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:03:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:37 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:37 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:03:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:37 smithi194 ceph-mon[32464]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:03:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:03:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:03:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-mon[29577]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:38.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:38.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:03:38.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:03:38.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:38.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:03:38.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:37 smithi119 ceph-mon[40674]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:38.404 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:03:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:03:38.974 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:38 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:39.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:38 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:39.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:38 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:39.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:39 smithi119 ceph-mon[29577]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:39.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:39 smithi119 ceph-mon[40674]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:39.868 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:39 smithi194 ceph-mon[32464]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:40.473 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:03:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:03:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:03:41.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:40 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:41.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:40 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:41.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:40 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:41.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:40 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:41.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:40 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:03:41.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:40 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:03:41.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:40 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:41.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:40 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:41.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:40 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:41.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:40 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:41.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:40 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:41.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:40 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:03:41.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:40 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:03:41.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:40 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:41.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:40 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:41.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:40 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:41.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:40 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:41.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:40 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:41.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:40 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:03:41.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:40 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:03:41.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:40 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:03:42.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:41 smithi194 ceph-mon[32464]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:42.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:41 smithi119 ceph-mon[29577]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:42.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:41 smithi119 ceph-mon[40674]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:43.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:03:42] "GET /metrics HTTP/1.1" 200 219841 "" "Prometheus/2.43.0" 2024-09-10T19:03:44.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:44 smithi194 ceph-mon[32464]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:44.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:44 smithi119 ceph-mon[29577]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:44.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:44 smithi119 ceph-mon[40674]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:46.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:46 smithi194 ceph-mon[32464]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:46.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:46 smithi119 ceph-mon[29577]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:46.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:46 smithi119 ceph-mon[40674]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:47.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:47.468 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:47.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:03:48.483 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:48 smithi119 ceph-mon[40674]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:48.484 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:48 smithi119 ceph-mon[29577]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:48.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:48 smithi194 ceph-mon[32464]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:48.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:03:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:03:49.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:49 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:49.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:49 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:49.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:49 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:50.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:50 smithi194 ceph-mon[32464]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:50.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:03:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:03:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:03:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:50 smithi119 ceph-mon[29577]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:50.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:50 smithi119 ceph-mon[40674]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:52.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:52 smithi194 ceph-mon[32464]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:52.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:52 smithi119 ceph-mon[29577]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:52.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:52 smithi119 ceph-mon[40674]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:53.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:03:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:03:52] "GET /metrics HTTP/1.1" 200 219829 "" "Prometheus/2.43.0" 2024-09-10T19:03:54.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:54 smithi194 ceph-mon[32464]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:54.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:54 smithi119 ceph-mon[29577]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:54.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:54 smithi119 ceph-mon[40674]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:03:56.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:56 smithi194 ceph-mon[32464]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:56.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:56 smithi119 ceph-mon[40674]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:56 smithi119 ceph-mon[29577]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:58.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:58 smithi194 ceph-mon[32464]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:58.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:03:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:03:58.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:58 smithi119 ceph-mon[29577]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:58.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:58 smithi119 ceph-mon[40674]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:03:59.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:03:59 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:59.531 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:03:59 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:03:59.531 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:03:59 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:00.239 INFO:tasks.workunit.client.0.smithi119.stderr:Note: switching to 'fad5356a94e6044ce1062d748daa5a0f2ed53a7d'. 2024-09-10T19:04:00.239 INFO:tasks.workunit.client.0.smithi119.stderr: 2024-09-10T19:04:00.239 INFO:tasks.workunit.client.0.smithi119.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-10T19:04:00.239 INFO:tasks.workunit.client.0.smithi119.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-10T19:04:00.240 INFO:tasks.workunit.client.0.smithi119.stderr:state without impacting any branches by switching back to a branch. 2024-09-10T19:04:00.240 INFO:tasks.workunit.client.0.smithi119.stderr: 2024-09-10T19:04:00.240 INFO:tasks.workunit.client.0.smithi119.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-10T19:04:00.240 INFO:tasks.workunit.client.0.smithi119.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-10T19:04:00.240 INFO:tasks.workunit.client.0.smithi119.stderr: 2024-09-10T19:04:00.240 INFO:tasks.workunit.client.0.smithi119.stderr: git switch -c 2024-09-10T19:04:00.240 INFO:tasks.workunit.client.0.smithi119.stderr: 2024-09-10T19:04:00.240 INFO:tasks.workunit.client.0.smithi119.stderr:Or undo this operation with: 2024-09-10T19:04:00.240 INFO:tasks.workunit.client.0.smithi119.stderr: 2024-09-10T19:04:00.240 INFO:tasks.workunit.client.0.smithi119.stderr: git switch - 2024-09-10T19:04:00.240 INFO:tasks.workunit.client.0.smithi119.stderr: 2024-09-10T19:04:00.240 INFO:tasks.workunit.client.0.smithi119.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-10T19:04:00.240 INFO:tasks.workunit.client.0.smithi119.stderr: 2024-09-10T19:04:00.240 INFO:tasks.workunit.client.0.smithi119.stderr:HEAD is now at fad5356a94e qa/upgrade: change pacific/octopus upgrade tests to use published image 2024-09-10T19:04:00.244 DEBUG:teuthology.orchestra.run.smithi119:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-10T19:04:00.278 INFO:tasks.workunit.client.0.smithi119.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-10T19:04:00.280 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-10T19:04:00.281 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-10T19:04:00.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:00 smithi194 ceph-mon[32464]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:00.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:04:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:04:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:04:00.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:00 smithi119 ceph-mon[29577]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:00.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:00 smithi119 ceph-mon[40674]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:00.930 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-10T19:04:01.001 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-10T19:04:01.040 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-10T19:04:01.042 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-10T19:04:01.042 INFO:tasks.workunit.client.0.smithi119.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-10T19:04:01.081 INFO:tasks.workunit.client.0.smithi119.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-10T19:04:01.084 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-10T19:04:01.084 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-10T19:04:01.140 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-10T19:04:01.141 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-10T19:04:01.142 DEBUG:teuthology.orchestra.run.smithi119:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fad5356a94e6044ce1062d748daa5a0f2ed53a7d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-10T19:04:01.220 INFO:tasks.workunit.client.0.smithi119.stderr:+ parallel=1 2024-09-10T19:04:01.220 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' '' = --serial ']' 2024-09-10T19:04:01.220 INFO:tasks.workunit.client.0.smithi119.stderr:+ color= 2024-09-10T19:04:01.220 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -t 1 ']' 2024-09-10T19:04:01.220 INFO:tasks.workunit.client.0.smithi119.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-10T19:04:01.220 INFO:tasks.workunit.client.0.smithi119.stderr:+ declare -A pids 2024-09-10T19:04:01.220 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.220 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.220 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_aio 2024-09-10T19:04:01.221 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_aio' 2024-09-10T19:04:01.221 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_aio 2024-09-10T19:04:01.221 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.222 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_aio 2024-09-10T19:04:01.222 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79410 2024-09-10T19:04:01.222 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_aio on pid 79410' 2024-09-10T19:04:01.222 INFO:tasks.workunit.client.0.smithi119.stdout:test api_aio on pid 79410 2024-09-10T19:04:01.225 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79410 2024-09-10T19:04:01.225 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.225 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.226 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-10T19:04:01.226 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_aio_pp 2024-09-10T19:04:01.226 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_aio_pp' 2024-09-10T19:04:01.226 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_aio_pp 2024-09-10T19:04:01.226 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.226 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.226 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.226 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.226 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.226 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.226 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.226 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.226 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.227 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.228 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.228 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.228 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_aio_pp 2024-09-10T19:04:01.228 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79417 2024-09-10T19:04:01.228 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_aio_pp on pid 79417' 2024-09-10T19:04:01.228 INFO:tasks.workunit.client.0.smithi119.stdout:test api_aio_pp on pid 79417 2024-09-10T19:04:01.231 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79417 2024-09-10T19:04:01.232 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.232 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.232 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-10T19:04:01.232 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_io 2024-09-10T19:04:01.232 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_io' 2024-09-10T19:04:01.232 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_io 2024-09-10T19:04:01.232 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.232 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.232 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.232 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.232 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.232 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.232 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.233 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.233 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.233 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.233 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.233 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.233 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.233 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.233 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.233 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.233 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.233 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.233 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.233 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.233 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.234 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.235 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.235 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.235 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.235 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.235 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.235 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.235 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.235 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.235 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.235 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.235 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_io 2024-09-10T19:04:01.235 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79426 2024-09-10T19:04:01.235 INFO:tasks.workunit.client.0.smithi119.stdout:test api_io on pid 79426 2024-09-10T19:04:01.239 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_io on pid 79426' 2024-09-10T19:04:01.239 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79426 2024-09-10T19:04:01.239 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.239 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-10T19:04:01.239 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.239 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_io_pp 2024-09-10T19:04:01.239 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_io_pp' 2024-09-10T19:04:01.239 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_io_pp 2024-09-10T19:04:01.240 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.240 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.240 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.240 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.240 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.240 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.240 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.240 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.240 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.240 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.240 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.240 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.240 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.240 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.241 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.241 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.241 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.241 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.241 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.241 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.241 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.241 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.242 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.242 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.242 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.242 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.242 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.242 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.242 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.242 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.242 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.242 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.242 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.242 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.242 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.242 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.243 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.243 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.243 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.243 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.243 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.243 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.243 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.243 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.243 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.243 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.243 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.243 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.243 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.243 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.244 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.244 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.244 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.244 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.244 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.244 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_io_pp 2024-09-10T19:04:01.244 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79437 2024-09-10T19:04:01.244 INFO:tasks.workunit.client.0.smithi119.stdout:test api_io_pp on pid 79437 2024-09-10T19:04:01.250 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_io_pp on pid 79437' 2024-09-10T19:04:01.250 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79437 2024-09-10T19:04:01.250 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-10T19:04:01.250 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.250 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.250 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_asio 2024-09-10T19:04:01.251 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_asio' 2024-09-10T19:04:01.251 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_asio 2024-09-10T19:04:01.251 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.251 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.251 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.251 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.251 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.251 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.251 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.251 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.251 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.251 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.251 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.251 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.252 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.252 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.252 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.252 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.252 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.252 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.252 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.252 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.252 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.252 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.252 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.252 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.252 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.253 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.253 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.253 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.253 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.253 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.253 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.253 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.253 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.253 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.253 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.253 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.253 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.254 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.254 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.254 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.254 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.254 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.254 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.254 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.254 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.254 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.254 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.254 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.254 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.254 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.254 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.255 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.255 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.255 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.255 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.255 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.255 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.255 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.255 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.255 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.255 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.255 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.255 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.255 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.256 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.256 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.256 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.256 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.256 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.256 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.256 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.256 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.256 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.256 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.256 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.256 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.256 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.257 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.257 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_asio 2024-09-10T19:04:01.257 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.257 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79453 2024-09-10T19:04:01.257 INFO:tasks.workunit.client.0.smithi119.stdout:test api_asio on pid 79453 2024-09-10T19:04:01.265 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_asio on pid 79453' 2024-09-10T19:04:01.265 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79453 2024-09-10T19:04:01.265 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.265 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.265 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-10T19:04:01.265 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_list 2024-09-10T19:04:01.265 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_list' 2024-09-10T19:04:01.265 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_list 2024-09-10T19:04:01.265 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.266 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.267 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.267 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.267 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.267 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.267 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.267 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.267 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.267 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.267 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.267 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.267 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.267 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.267 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.267 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.268 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.268 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.268 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.268 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.268 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.268 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.268 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.268 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.268 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.268 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.268 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.268 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.269 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.269 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.269 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.269 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.269 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.269 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.269 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.269 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.269 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.269 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.269 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.269 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.269 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.269 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.270 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.270 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.270 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.270 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.270 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.270 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.270 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.270 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.270 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.270 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.270 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.270 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.270 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.270 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79410/exe 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.271 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.272 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.272 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.272 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.272 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.272 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.272 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.272 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.272 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.272 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.272 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.272 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.272 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.272 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.273 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.273 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.273 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.273 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_list 2024-09-10T19:04:01.273 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.273 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79470 2024-09-10T19:04:01.273 INFO:tasks.workunit.client.0.smithi119.stdout:test api_list on pid 79470 2024-09-10T19:04:01.276 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_list on pid 79470' 2024-09-10T19:04:01.276 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79470 2024-09-10T19:04:01.276 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.276 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.276 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-10T19:04:01.276 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.276 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_lock 2024-09-10T19:04:01.276 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_lock' 2024-09-10T19:04:01.276 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.276 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.277 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.277 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.277 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.277 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.277 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.277 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.277 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.277 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.277 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.277 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_lock 2024-09-10T19:04:01.278 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_aio 2024-09-10T19:04:01.278 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.278 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.278 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.278 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.278 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.278 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.278 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.278 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.278 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.278 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.278 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.278 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.278 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.279 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.279 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.279 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.279 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.279 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-10T19:04:01.279 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.279 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.279 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.279 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.279 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.279 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.279 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.279 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.279 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.280 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.280 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.280 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.280 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.280 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.280 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.280 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.280 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.280 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.280 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.280 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.280 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.281 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.281 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.281 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.281 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.281 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.281 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.281 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.281 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.281 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.281 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.281 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.281 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_aio: /' 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.282 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.283 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.283 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.283 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.283 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.283 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.283 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.283 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.283 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.283 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.283 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.283 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.283 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.283 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.284 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.284 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.284 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.284 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.284 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.284 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.284 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.284 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.284 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.284 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.284 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.284 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.284 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.284 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.285 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.285 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.285 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79417/exe 2024-09-10T19:04:01.285 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.285 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.285 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.285 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.285 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.285 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_lock 2024-09-10T19:04:01.286 INFO:tasks.workunit.client.0.smithi119.stdout:test api_lock on pid 79490 2024-09-10T19:04:01.287 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79490 2024-09-10T19:04:01.287 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_lock on pid 79490' 2024-09-10T19:04:01.287 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79490 2024-09-10T19:04:01.287 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.287 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.287 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.288 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-10T19:04:01.288 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_lock_pp 2024-09-10T19:04:01.288 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.288 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_lock_pp' 2024-09-10T19:04:01.288 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.288 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.288 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.288 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.288 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.288 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.288 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.288 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.288 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.289 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.289 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.289 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.289 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.289 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.289 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.289 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.289 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.289 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.289 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.289 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.289 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.290 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.291 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.291 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.291 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.291 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.291 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.291 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.291 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.291 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.291 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.291 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.291 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.291 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.291 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.292 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.292 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_lock_pp 2024-09-10T19:04:01.292 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.292 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.292 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.292 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.292 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.292 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.292 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.292 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.292 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.292 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.292 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.292 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.293 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.293 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.293 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.293 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.293 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.293 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.293 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.293 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.293 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.293 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.293 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.293 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_aio_pp 2024-09-10T19:04:01.293 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.294 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.294 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.294 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.294 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.294 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.294 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.294 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.294 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.294 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.294 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.294 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.294 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.294 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.294 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.295 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.296 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79426/exe 2024-09-10T19:04:01.296 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-10T19:04:01.296 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-10T19:04:01.296 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.296 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.296 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.296 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.296 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.296 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.296 INFO:tasks.workunit.client.0.smithi119.stdout:test api_lock_pp on pid 79510 2024-09-10T19:04:01.300 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_lock_pp 2024-09-10T19:04:01.300 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79510 2024-09-10T19:04:01.301 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_lock_pp on pid 79510' 2024-09-10T19:04:01.301 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79510 2024-09-10T19:04:01.301 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.301 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.301 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.301 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-10T19:04:01.301 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.301 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.301 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.301 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.301 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.301 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.301 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.302 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.302 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.302 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.302 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.302 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_io.log 2024-09-10T19:04:01.302 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.302 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.302 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.302 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.302 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.302 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.302 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.302 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.302 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.303 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.303 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.303 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.303 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.303 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.303 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.303 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.303 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.303 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.303 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.303 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.303 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.304 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.304 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.304 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.304 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.304 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.304 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.304 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.304 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.304 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.304 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.304 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.304 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.304 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.304 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.305 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.305 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.305 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.305 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.305 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.305 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.305 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.305 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.305 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.305 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.305 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_io: /' 2024-09-10T19:04:01.305 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.305 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.306 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.306 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.306 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.306 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.306 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.306 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.306 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.306 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.306 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.306 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.306 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.306 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.306 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.306 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.307 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.307 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.307 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.307 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.307 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.307 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.307 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.307 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.307 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.307 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.307 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.307 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.308 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.308 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.308 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.308 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.308 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.308 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.308 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.308 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.308 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.308 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.308 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.308 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.308 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.308 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.309 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.309 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.309 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.309 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.309 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_misc 2024-09-10T19:04:01.309 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.309 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_misc' 2024-09-10T19:04:01.309 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.309 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.309 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.309 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.309 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.309 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_io 2024-09-10T19:04:01.310 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.310 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79437/exe 2024-09-10T19:04:01.310 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.310 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.310 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.310 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.310 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.310 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.310 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.310 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.310 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.310 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.311 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.311 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.311 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.311 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.311 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.311 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.311 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.311 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.311 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.311 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.311 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.311 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.311 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.311 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.312 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.312 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.312 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.312 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.312 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.312 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.312 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.312 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.312 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.312 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.312 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.312 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.312 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.313 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.313 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.313 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.313 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.313 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.313 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.313 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.313 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.313 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.313 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.313 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.313 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.313 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.314 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.314 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.314 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.314 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.314 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.314 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.314 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.314 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.314 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.314 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.314 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.314 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.314 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.314 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79453/exe 2024-09-10T19:04:01.315 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.316 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.316 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_misc 2024-09-10T19:04:01.316 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-10T19:04:01.316 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.316 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-10T19:04:01.316 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_io_pp 2024-09-10T19:04:01.316 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.316 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.316 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.316 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.316 INFO:tasks.workunit.client.0.smithi119.stdout:test api_misc on pid 79541 2024-09-10T19:04:01.319 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.320 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.320 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.320 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_misc 2024-09-10T19:04:01.320 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.320 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.320 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.320 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79541 2024-09-10T19:04:01.320 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_misc on pid 79541' 2024-09-10T19:04:01.320 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79541 2024-09-10T19:04:01.320 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.320 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.320 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-10T19:04:01.321 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.321 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_asio: /' 2024-09-10T19:04:01.321 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_misc_pp 2024-09-10T19:04:01.321 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_asio 2024-09-10T19:04:01.321 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_misc_pp' 2024-09-10T19:04:01.321 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-10T19:04:01.321 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.321 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.321 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.321 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.321 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.321 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.321 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.322 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.322 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.322 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.322 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.322 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.322 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.322 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.322 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.322 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.322 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.322 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.322 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.322 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.322 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.323 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.323 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.323 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.323 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.323 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.323 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.323 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.323 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.323 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.323 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.323 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.323 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.323 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.323 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.324 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.324 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.324 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.324 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.324 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.324 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.324 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.324 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.324 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.324 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.324 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.324 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.324 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.324 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.325 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.325 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.325 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.325 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.325 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.325 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.325 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.325 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.325 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.325 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.325 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.325 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.325 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.326 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.327 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.327 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.327 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.327 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.327 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.327 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.327 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.327 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.327 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.327 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.327 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.327 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.327 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.327 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.328 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.328 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.328 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.328 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.328 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.328 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.328 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.328 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.328 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.328 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.328 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.328 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.328 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.328 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.329 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.329 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.329 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.329 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.329 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.329 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.329 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.329 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.329 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.329 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.329 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.329 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.329 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.329 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.330 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.330 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.330 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.330 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.330 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_misc_pp 2024-09-10T19:04:01.330 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.330 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.330 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.330 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.330 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.330 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79490/exe 2024-09-10T19:04:01.330 INFO:tasks.workunit.client.0.smithi119.stdout:test api_misc_pp on pid 79567 2024-09-10T19:04:01.332 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_misc_pp 2024-09-10T19:04:01.332 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79567 2024-09-10T19:04:01.333 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_misc_pp on pid 79567' 2024-09-10T19:04:01.333 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79567 2024-09-10T19:04:01.333 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.333 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.333 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.333 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.333 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.333 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.333 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.333 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.333 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.333 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.333 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.333 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_tier_pp 2024-09-10T19:04:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_tier_pp' 2024-09-10T19:04:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.334 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.335 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_lock 2024-09-10T19:04:01.335 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.335 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.335 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.335 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.335 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.335 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.335 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.335 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.335 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.335 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.335 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.335 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_lock: /' 2024-09-10T19:04:01.336 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_tier_pp 2024-09-10T19:04:01.337 INFO:tasks.workunit.client.0.smithi119.stdout:test api_tier_pp on pid 79578 2024-09-10T19:04:01.340 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_tier_pp 2024-09-10T19:04:01.340 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79578 2024-09-10T19:04:01.340 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_tier_pp on pid 79578' 2024-09-10T19:04:01.340 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79578 2024-09-10T19:04:01.340 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.340 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.340 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_pool 2024-09-10T19:04:01.340 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_pool' 2024-09-10T19:04:01.340 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-10T19:04:01.340 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-10T19:04:01.341 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_pool 2024-09-10T19:04:01.341 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.341 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.341 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.341 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.341 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.341 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.341 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.341 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.341 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.341 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.341 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.341 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.342 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.342 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79510/exe 2024-09-10T19:04:01.342 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.342 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.342 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.342 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.342 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.342 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.342 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.342 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.342 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-10T19:04:01.342 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.342 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.342 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.343 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.344 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.344 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.344 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.344 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.344 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.344 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.344 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.344 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.344 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.344 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.344 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.344 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.344 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.344 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.345 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.345 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.345 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.345 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.345 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.345 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.345 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.345 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.345 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.345 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.345 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.345 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.345 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.345 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.346 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.346 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.346 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.346 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.346 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.346 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.346 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.346 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.346 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.346 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.346 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.346 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.346 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.346 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.347 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.347 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.347 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.347 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.347 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.347 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.347 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.347 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.347 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.347 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.347 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.347 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.347 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.347 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.348 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.348 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.348 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.348 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.348 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.348 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.348 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.348 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.348 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.348 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.348 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.348 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.348 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79470/exe 2024-09-10T19:04:01.348 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.349 INFO:tasks.workunit.client.0.smithi119.stdout:test api_pool on pid 79617 2024-09-10T19:04:01.351 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_pool 2024-09-10T19:04:01.351 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79617 2024-09-10T19:04:01.351 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_pool on pid 79617' 2024-09-10T19:04:01.351 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79617 2024-09-10T19:04:01.351 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.351 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.351 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.351 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_lock_pp 2024-09-10T19:04:01.352 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_snapshots 2024-09-10T19:04:01.352 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_snapshots' 2024-09-10T19:04:01.352 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.352 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.352 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.352 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.352 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.352 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-10T19:04:01.352 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.352 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.352 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.352 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.353 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.353 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.353 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.353 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.353 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.353 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.353 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.353 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.353 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.353 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.353 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.353 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.353 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.354 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.354 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.354 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.354 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.354 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.354 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.354 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.354 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.354 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_list 2024-09-10T19:04:01.354 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.354 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.354 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.354 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.354 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.355 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_list: /' 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_list.log 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.356 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.357 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.357 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.357 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.357 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.357 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.357 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.357 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.357 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.357 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.357 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.357 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.357 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.358 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.358 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.358 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.358 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.358 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.358 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.358 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.358 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.358 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.358 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.358 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.358 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.358 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.358 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.359 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.359 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.359 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.359 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.359 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.359 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.359 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.359 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.359 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.359 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.359 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_snapshots 2024-09-10T19:04:01.360 INFO:tasks.workunit.client.0.smithi119.stdout:test api_snapshots on pid 79666 2024-09-10T19:04:01.361 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_snapshots 2024-09-10T19:04:01.362 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79666 2024-09-10T19:04:01.362 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_snapshots on pid 79666' 2024-09-10T19:04:01.362 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79666 2024-09-10T19:04:01.362 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.362 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.362 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79541/exe 2024-09-10T19:04:01.362 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-10T19:04:01.362 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.362 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.362 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.362 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.362 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.362 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.363 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.363 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.363 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.363 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.363 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.363 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.363 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.363 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.363 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.363 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.363 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.363 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.363 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.364 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.364 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.364 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.364 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.364 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.364 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.364 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.364 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.364 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.364 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.364 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.364 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.364 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.365 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.365 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.365 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.365 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.365 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.365 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.365 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.365 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.365 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.365 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.365 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.365 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.365 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.366 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_misc 2024-09-10T19:04:01.366 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_misc: /' 2024-09-10T19:04:01.366 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-10T19:04:01.366 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.366 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.366 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.366 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.366 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.366 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.366 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.366 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.366 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.366 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.366 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.367 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_snapshots_pp 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_snapshots_pp' 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_snapshots_pp 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.368 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.369 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.369 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.369 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.369 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.369 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.369 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.369 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.369 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.369 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.369 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.370 INFO:tasks.workunit.client.0.smithi119.stdout:test api_snapshots_pp on pid 79701 2024-09-10T19:04:01.372 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_snapshots_pp 2024-09-10T19:04:01.372 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79701 2024-09-10T19:04:01.372 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_snapshots_pp on pid 79701' 2024-09-10T19:04:01.372 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79701 2024-09-10T19:04:01.372 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.372 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.372 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-10T19:04:01.372 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.373 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.373 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.373 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_stat 2024-09-10T19:04:01.373 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_stat' 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_stat 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.374 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.375 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.376 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.376 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.376 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.376 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.376 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.376 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.376 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.376 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.376 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.376 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.376 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.376 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.376 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.376 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.377 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.378 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_stat 2024-09-10T19:04:01.378 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79721 2024-09-10T19:04:01.378 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_stat on pid 79721' 2024-09-10T19:04:01.378 INFO:tasks.workunit.client.0.smithi119.stdout:test api_stat on pid 79721 2024-09-10T19:04:01.380 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79721 2024-09-10T19:04:01.380 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.381 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.381 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-10T19:04:01.381 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.381 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.381 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.381 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.381 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.381 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_stat_pp 2024-09-10T19:04:01.381 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_stat_pp' 2024-09-10T19:04:01.381 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.381 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.381 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.382 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.382 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.382 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.382 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.382 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.382 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.382 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.382 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.382 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.382 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_stat_pp 2024-09-10T19:04:01.382 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.382 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.382 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.382 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.383 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.383 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.383 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.383 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.383 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.383 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.383 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.383 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.383 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.383 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.383 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.383 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.383 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.384 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.385 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.385 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.385 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.385 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.385 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.385 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.385 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.385 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.385 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.385 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.385 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.385 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.385 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.386 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.387 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.387 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.387 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.387 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.387 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.387 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.387 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.387 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.387 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.387 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.387 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.387 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.387 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.388 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.388 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.388 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.388 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.388 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.388 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.388 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.388 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.388 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.388 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.388 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.388 INFO:tasks.workunit.client.0.smithi119.stdout:test api_stat_pp on pid 79753 2024-09-10T19:04:01.395 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_stat_pp 2024-09-10T19:04:01.395 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79753 2024-09-10T19:04:01.396 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_stat_pp on pid 79753' 2024-09-10T19:04:01.396 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79753 2024-09-10T19:04:01.396 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.396 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.396 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.396 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.396 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.396 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.396 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.396 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.396 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.396 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.396 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.396 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.397 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.397 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.397 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.397 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.397 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.397 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.397 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.397 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.397 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.397 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.397 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.397 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.397 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.398 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.398 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.398 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.398 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.398 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.398 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.398 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.398 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.398 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.398 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.398 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.398 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.398 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.399 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.399 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.399 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.399 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.399 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.399 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.399 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.399 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.399 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-10T19:04:01.399 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.399 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.399 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.399 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.399 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.400 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.401 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.401 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.401 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.401 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.401 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.401 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.401 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.401 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.401 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.401 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.401 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.401 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.401 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.401 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.402 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.402 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.402 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.402 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.402 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.402 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.402 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79578/exe 2024-09-10T19:04:01.402 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.402 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.402 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.402 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.402 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.402 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.402 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.403 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79617/exe 2024-09-10T19:04:01.403 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.403 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.403 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.403 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.403 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.403 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.403 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.403 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.403 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.403 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.403 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.403 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.404 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.404 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.404 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.404 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.404 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.404 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.404 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.404 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.404 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.404 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.404 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.404 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.404 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.405 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.405 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.405 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.405 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.405 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.405 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.405 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.405 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.405 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.405 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.405 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.405 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.406 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.406 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.406 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.406 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.406 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.406 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.406 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.406 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.406 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.406 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.406 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.406 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.406 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.406 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.407 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.407 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.407 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.407 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.407 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.407 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.407 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.407 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.407 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.407 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.407 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.407 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.407 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_pool: /' 2024-09-10T19:04:01.407 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_pool 2024-09-10T19:04:01.408 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.408 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79567/exe 2024-09-10T19:04:01.408 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-10T19:04:01.408 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.408 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.408 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.408 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.408 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.408 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.408 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.408 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.408 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.408 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.409 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.409 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.409 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.409 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-10T19:04:01.409 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.409 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.409 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.409 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.409 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.409 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.409 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.409 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.409 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_tier_pp 2024-09-10T19:04:01.409 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.410 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_watch_notify 2024-09-10T19:04:01.410 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.410 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.410 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.410 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.410 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.410 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_watch_notify' 2024-09-10T19:04:01.410 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.410 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.410 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.410 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.410 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.410 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.410 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.411 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.411 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.411 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.411 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.411 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.411 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-10T19:04:01.411 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.411 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.411 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.411 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.411 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.411 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.411 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.412 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.412 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.412 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.412 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.412 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.412 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.412 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.412 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.412 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.412 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.412 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.412 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.412 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.412 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.413 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.413 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.413 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.413 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.413 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.413 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.413 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.413 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.413 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.413 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.413 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.413 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.413 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.413 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.414 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.414 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.414 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.414 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.414 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.414 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.414 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.414 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.414 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.414 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-10T19:04:01.414 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79666/exe 2024-09-10T19:04:01.414 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.414 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-10T19:04:01.415 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_misc_pp 2024-09-10T19:04:01.415 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_watch_notify 2024-09-10T19:04:01.415 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.415 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79701/exe 2024-09-10T19:04:01.415 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.415 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.415 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.415 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.415 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.415 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.415 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.415 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.415 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.415 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.416 INFO:tasks.workunit.client.0.smithi119.stdout:test api_watch_notify on pid 79827 2024-09-10T19:04:01.418 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_watch_notify 2024-09-10T19:04:01.418 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79827 2024-09-10T19:04:01.418 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_watch_notify on pid 79827' 2024-09-10T19:04:01.419 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79827 2024-09-10T19:04:01.419 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.419 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.419 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-10T19:04:01.419 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_watch_notify_pp 2024-09-10T19:04:01.419 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_snapshots 2024-09-10T19:04:01.419 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-10T19:04:01.419 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-10T19:04:01.419 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.419 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_watch_notify_pp' 2024-09-10T19:04:01.419 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.419 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.419 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.419 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.420 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.420 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.420 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.420 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.420 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.420 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.420 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.420 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.420 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.420 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.420 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.420 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.420 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.421 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.421 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.421 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.421 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.421 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.421 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.421 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.421 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.421 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.421 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.421 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.421 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.422 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.422 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.422 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.422 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.422 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.422 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.422 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.422 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.422 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.422 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.422 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.422 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.422 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.423 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.423 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.423 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.423 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.423 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.423 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.423 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.423 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.423 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.423 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.423 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.423 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.423 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.424 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.425 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.425 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.425 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.425 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.425 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-10T19:04:01.425 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-10T19:04:01.425 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_snapshots_pp 2024-09-10T19:04:01.425 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_watch_notify_pp 2024-09-10T19:04:01.425 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_watch_notify_pp 2024-09-10T19:04:01.425 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79849 2024-09-10T19:04:01.425 INFO:tasks.workunit.client.0.smithi119.stdout:test api_watch_notify_pp on pid 79849 2024-09-10T19:04:01.428 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_watch_notify_pp on pid 79849' 2024-09-10T19:04:01.429 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79849 2024-09-10T19:04:01.429 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.429 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.429 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_cmd 2024-09-10T19:04:01.429 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_cmd' 2024-09-10T19:04:01.429 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_cmd 2024-09-10T19:04:01.429 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.429 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.429 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.429 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.429 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.429 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.430 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.430 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.430 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.430 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.430 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.430 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.430 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.430 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.430 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.430 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.430 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.430 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.430 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.431 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.432 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.432 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.432 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.432 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.432 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.432 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.432 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.432 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.432 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.432 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.432 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.432 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.432 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.432 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79721/exe 2024-09-10T19:04:01.433 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.433 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.433 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-10T19:04:01.433 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.433 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.433 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.433 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.433 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.433 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.433 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.433 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.433 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.433 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.433 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_stat 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_stat: /' 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.434 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.435 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.435 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.435 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.435 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.435 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.435 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.435 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.435 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.435 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.435 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.435 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.435 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.435 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.435 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.436 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.436 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.436 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.436 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.436 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.436 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.436 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.436 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.436 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.436 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.436 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.436 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.436 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.437 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.437 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.437 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.437 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.437 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.437 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.437 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.437 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.437 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.437 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.437 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.437 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.437 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.437 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.438 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.438 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.438 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.438 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.438 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.438 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.438 INFO:tasks.workunit.client.0.smithi119.stdout:test api_cmd on pid 79892 2024-09-10T19:04:01.441 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_cmd 2024-09-10T19:04:01.441 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79892 2024-09-10T19:04:01.441 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_cmd on pid 79892' 2024-09-10T19:04:01.441 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79892 2024-09-10T19:04:01.441 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.441 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_cmd_pp 2024-09-10T19:04:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_cmd_pp' 2024-09-10T19:04:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-10T19:04:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.442 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.443 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.443 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.443 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.443 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.443 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.443 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_cmd_pp 2024-09-10T19:04:01.443 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.443 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.443 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.443 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.443 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.443 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.443 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.443 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.444 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.444 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.444 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.444 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.444 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.444 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.444 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.444 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.444 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.444 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.444 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.444 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.445 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.446 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.446 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.446 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.446 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.446 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.446 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.446 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.446 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.446 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.446 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.446 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.446 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.446 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_cmd_pp 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79929 2024-09-10T19:04:01.447 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_cmd_pp on pid 79929' 2024-09-10T19:04:01.448 INFO:tasks.workunit.client.0.smithi119.stdout:test api_cmd_pp on pid 79929 2024-09-10T19:04:01.452 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79929 2024-09-10T19:04:01.452 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.452 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.453 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.453 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.453 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.453 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.453 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.453 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.453 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.453 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.453 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.453 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.453 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.453 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.453 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.454 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.454 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.454 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.454 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.454 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.454 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.454 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.454 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.454 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_service 2024-09-10T19:04:01.454 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.454 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.454 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.455 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.455 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.455 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.455 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.455 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.455 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_service' 2024-09-10T19:04:01.455 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.455 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.455 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.455 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.455 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.455 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.456 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.456 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.456 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.456 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.456 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.456 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.456 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79827/exe 2024-09-10T19:04:01.456 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.456 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-10T19:04:01.456 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.456 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.456 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.456 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.457 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.457 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.457 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.457 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.457 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.457 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.457 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.457 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.457 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.457 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.457 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.457 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.457 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.458 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.458 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.458 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-10T19:04:01.458 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.458 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.458 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.458 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.458 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.458 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.458 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.458 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.458 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.458 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.458 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.459 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.459 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.459 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.459 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.459 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-10T19:04:01.459 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.459 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.459 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.459 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.459 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.459 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.459 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.459 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.459 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.460 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.461 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.462 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.462 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.462 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.462 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_watch_notify 2024-09-10T19:04:01.462 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_service 2024-09-10T19:04:01.462 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.462 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.462 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.462 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.462 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.462 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.462 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.462 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.462 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79753/exe 2024-09-10T19:04:01.463 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.463 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.463 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.463 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.463 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.463 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.463 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.463 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.463 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.463 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.463 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.463 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.463 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.464 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.464 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.464 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.464 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.464 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.464 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.464 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.464 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.464 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.464 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.464 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.464 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.464 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.465 INFO:tasks.workunit.client.0.smithi119.stdout:test api_service on pid 79996 2024-09-10T19:04:01.468 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.468 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_service 2024-09-10T19:04:01.468 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79996 2024-09-10T19:04:01.468 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_service on pid 79996' 2024-09-10T19:04:01.468 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=79996 2024-09-10T19:04:01.468 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.468 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.468 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.468 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.468 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.468 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.469 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.469 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.469 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.469 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.469 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.469 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.469 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.469 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.469 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.469 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.469 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.469 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.469 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.470 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.470 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.470 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.470 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.470 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_service_pp 2024-09-10T19:04:01.470 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_service_pp' 2024-09-10T19:04:01.470 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-10T19:04:01.470 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.470 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.470 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.470 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.470 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.470 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.470 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.471 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.471 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.471 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.471 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.471 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.471 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.471 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.471 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.471 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-10T19:04:01.471 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_stat_pp 2024-09-10T19:04:01.471 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.471 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.471 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-10T19:04:01.471 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.472 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.472 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.472 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.472 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.472 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.472 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.472 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.472 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.472 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.472 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.472 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.472 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.472 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.473 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.473 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.473 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.473 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.473 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.473 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.473 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.473 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.473 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.473 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.473 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.473 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.473 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.473 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.474 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.474 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.474 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.474 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.474 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.474 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.474 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.474 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.474 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.474 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.474 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.474 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.474 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.475 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.475 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.475 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.475 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.475 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.475 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.475 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79892/exe 2024-09-10T19:04:01.475 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.475 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.475 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.475 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_service_pp 2024-09-10T19:04:01.475 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.475 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.475 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.476 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.476 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.476 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.476 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.476 INFO:tasks.workunit.client.0.smithi119.stdout:test api_service_pp on pid 80039 2024-09-10T19:04:01.479 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.480 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.480 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.480 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.480 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.480 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.480 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.480 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.480 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.480 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.480 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.480 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_service_pp 2024-09-10T19:04:01.480 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.481 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.481 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.481 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.481 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.481 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.481 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.481 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.481 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.481 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.481 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=80039 2024-09-10T19:04:01.481 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.481 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.481 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_service_pp on pid 80039' 2024-09-10T19:04:01.482 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=80039 2024-09-10T19:04:01.482 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.482 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.482 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.482 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.482 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.482 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.482 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.482 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.482 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.482 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-10T19:04:01.482 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-10T19:04:01.482 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.483 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.483 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_cmd: /' 2024-09-10T19:04:01.483 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.483 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.483 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.483 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_cmd 2024-09-10T19:04:01.483 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_c_write_operations 2024-09-10T19:04:01.483 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_c_write_operations' 2024-09-10T19:04:01.483 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.483 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.483 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.483 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.483 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.483 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.484 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.484 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.484 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.484 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.484 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.484 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.484 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.484 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.484 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.484 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.484 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.484 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.484 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.485 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.485 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.485 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.485 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.485 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.485 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.485 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.485 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.485 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.485 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.485 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.485 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.485 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.485 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.486 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.486 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.486 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.486 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.486 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.486 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.486 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.486 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.486 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.486 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.486 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.486 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.486 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.487 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.487 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.487 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.487 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.487 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.487 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.487 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.487 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.487 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.487 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.487 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.487 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.487 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.488 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.488 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.488 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.488 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.488 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.488 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.488 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.488 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_c_write_operations 2024-09-10T19:04:01.488 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.488 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.488 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.488 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.488 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.489 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.489 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.489 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.489 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.489 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.489 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.489 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.489 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.489 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.489 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.489 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79849/exe 2024-09-10T19:04:01.489 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_c_write_operations 2024-09-10T19:04:01.490 INFO:tasks.workunit.client.0.smithi119.stdout:test api_c_write_operations on pid 80060 2024-09-10T19:04:01.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=80060 2024-09-10T19:04:01.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_c_write_operations on pid 80060' 2024-09-10T19:04:01.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=80060 2024-09-10T19:04:01.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.493 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-10T19:04:01.493 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.493 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.493 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.493 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.493 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.494 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.494 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.494 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.494 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.494 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_c_read_operations 2024-09-10T19:04:01.494 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.494 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.494 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.494 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.494 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.494 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.494 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.494 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.495 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.495 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.495 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.495 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.495 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.495 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.495 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.495 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.495 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.495 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.495 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.495 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.495 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_c_read_operations' 2024-09-10T19:04:01.496 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.496 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.496 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.496 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.496 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.496 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.496 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.496 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.496 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.496 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.496 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.496 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.496 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.496 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.497 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.497 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.497 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.497 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.497 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.497 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.497 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.497 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.497 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.497 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.497 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79929/exe 2024-09-10T19:04:01.497 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.497 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.498 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_c_read_operations 2024-09-10T19:04:01.498 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.498 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.498 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.498 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.498 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_watch_notify_pp 2024-09-10T19:04:01.498 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.498 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.498 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.498 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.498 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.498 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.498 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.498 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-10T19:04:01.499 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.499 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.499 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.499 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.499 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.499 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.499 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.499 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.499 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.499 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.500 INFO:tasks.workunit.client.0.smithi119.stdout:test api_c_read_operations on pid 80072 2024-09-10T19:04:01.502 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.502 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_c_read_operations 2024-09-10T19:04:01.502 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.502 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.502 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.502 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.502 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.502 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=80072 2024-09-10T19:04:01.502 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_c_read_operations on pid 80072' 2024-09-10T19:04:01.502 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.503 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=80072 2024-09-10T19:04:01.503 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.503 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.503 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.503 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.503 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.503 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.503 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.503 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.503 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s api_cls_remote_reads 2024-09-10T19:04:01.503 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' api_cls_remote_reads' 2024-09-10T19:04:01.503 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-10T19:04:01.503 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_cmd_pp 2024-09-10T19:04:01.503 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-10T19:04:01.504 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.504 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.504 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.504 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-10T19:04:01.504 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.504 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.504 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.504 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.504 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.504 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.504 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.504 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.505 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.505 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.505 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.505 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.505 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.505 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.505 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.505 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.505 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.505 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.505 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.505 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.505 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo api_cls_remote_reads 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=api_cls_remote_reads 2024-09-10T19:04:01.506 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-10T19:04:01.507 INFO:tasks.workunit.client.0.smithi119.stdout:test api_cls_remote_reads on pid 80087 2024-09-10T19:04:01.511 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=80087 2024-09-10T19:04:01.511 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test api_cls_remote_reads on pid 80087' 2024-09-10T19:04:01.512 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=80087 2024-09-10T19:04:01.512 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.512 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.512 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.512 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.512 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.512 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.512 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.512 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.512 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.512 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.512 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.512 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.513 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.513 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.513 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.513 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.513 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.513 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.513 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.513 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.513 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.513 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.513 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.513 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.513 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-09-10T19:04:01.513 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.514 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.514 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.514 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.514 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.514 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.514 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.514 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.514 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.514 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.514 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.514 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.514 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.514 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.515 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.515 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.515 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.515 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.515 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.515 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.515 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.515 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.515 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.515 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.515 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.515 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.516 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.516 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.516 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.516 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.516 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.516 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.516 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.516 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.516 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.516 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.516 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.516 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.516 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.516 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.517 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.517 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.517 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.517 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.517 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.517 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.517 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.517 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.517 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.517 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.517 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.517 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.517 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.517 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.518 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.518 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.518 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.518 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.518 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.518 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s list_parallel 2024-09-10T19:04:01.518 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.518 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' list_parallel' 2024-09-10T19:04:01.518 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.518 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.518 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.518 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.518 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.519 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.520 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.520 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.520 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.520 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.520 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.520 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.520 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.520 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.520 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.520 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.520 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.520 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.520 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.520 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo list_parallel 2024-09-10T19:04:01.521 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.522 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.522 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.522 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.522 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.522 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.522 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.522 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.522 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.522 INFO:tasks.workunit.client.0.smithi119.stdout:test list_parallel on pid 80124 2024-09-10T19:04:01.526 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=list_parallel 2024-09-10T19:04:01.526 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.526 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.526 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.526 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.526 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.527 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.527 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=80124 2024-09-10T19:04:01.527 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.527 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test list_parallel on pid 80124' 2024-09-10T19:04:01.527 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.527 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=80124 2024-09-10T19:04:01.527 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.527 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.527 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.527 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.527 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.527 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.527 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s open_pools_parallel 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.528 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' open_pools_parallel' 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.529 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.530 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/80039/exe 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.531 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.532 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.533 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.533 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.533 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.533 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.533 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.533 INFO:tasks.workunit.client.0.smithi119.stdout:test open_pools_parallel on pid 80155 2024-09-10T19:04:01.535 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.535 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.535 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.535 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo open_pools_parallel 2024-09-10T19:04:01.535 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.535 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.535 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=open_pools_parallel 2024-09-10T19:04:01.535 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.535 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.536 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.536 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.536 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.536 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.536 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=80155 2024-09-10T19:04:01.536 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.536 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test open_pools_parallel on pid 80155' 2024-09-10T19:04:01.536 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.536 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=80155 2024-09-10T19:04:01.536 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.536 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.536 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T19:04:01.536 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.536 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.537 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.537 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.537 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.537 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.537 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_service_pp 2024-09-10T19:04:01.537 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.537 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.537 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.537 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.537 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.537 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.537 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.537 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.537 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.538 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.538 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.538 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.538 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.538 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.538 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.538 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.538 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.538 INFO:tasks.workunit.client.0.smithi119.stderr:++ printf %25s delete_pools_parallel 2024-09-10T19:04:01.538 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.538 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.538 INFO:tasks.workunit.client.0.smithi119.stderr:+ r=' delete_pools_parallel' 2024-09-10T19:04:01.538 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-10T19:04:01.538 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-10T19:04:01.539 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-10T19:04:01.539 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.539 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.539 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.539 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.539 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.539 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.539 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.539 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.539 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.539 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.539 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.539 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.539 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:++ echo delete_pools_parallel 2024-09-10T19:04:01.540 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.541 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.541 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.541 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.541 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.541 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.541 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.541 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.541 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.541 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.541 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.541 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.541 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.541 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.542 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.542 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.542 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.542 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.542 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.542 INFO:tasks.workunit.client.0.smithi119.stderr:++ awk '{print $1}' 2024-09-10T19:04:01.542 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.542 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.542 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.542 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.542 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.542 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.542 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.542 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.543 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.543 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.543 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.543 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.543 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.543 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.543 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.543 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.543 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.543 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.543 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.543 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.543 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.543 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.544 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.544 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.544 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.544 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.544 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.544 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.544 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.544 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.544 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.544 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.544 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.544 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.544 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.544 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.545 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.545 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.545 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.545 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.545 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.545 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.545 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.545 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.545 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.545 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.545 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.545 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.545 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.546 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.546 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.546 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.546 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.546 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.546 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.546 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.546 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.546 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.546 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.546 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.546 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.546 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.546 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.547 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.547 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.547 INFO:tasks.workunit.client.0.smithi119.stdout:test delete_pools_parallel on pid 80185 2024-09-10T19:04:01.557 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.557 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.557 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.557 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.557 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.557 INFO:tasks.workunit.client.0.smithi119.stderr:+ ff=delete_pools_parallel 2024-09-10T19:04:01.557 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.557 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=80185 2024-09-10T19:04:01.558 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo 'test delete_pools_parallel on pid 80185' 2024-09-10T19:04:01.558 INFO:tasks.workunit.client.0.smithi119.stderr:+ pids[$f]=80185 2024-09-10T19:04:01.558 INFO:tasks.workunit.client.0.smithi119.stderr:+ ret=0 2024-09-10T19:04:01.558 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T19:04:01.558 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:04:01.558 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79578 2024-09-10T19:04:01.558 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79578 2024-09-10T19:04:01.558 INFO:tasks.workunit.client.0.smithi119.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-10T19:04:01.558 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/79996/exe 2024-09-10T19:04:01.558 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -f /etc/bashrc ']' 2024-09-10T19:04:01.558 INFO:tasks.workunit.client.0.smithi119.stderr:+ . /etc/bashrc 2024-09-10T19:04:01.558 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -z '' ']' 2024-09-10T19:04:01.558 INFO:tasks.workunit.client.0.smithi119.stderr:++ BASHRCSOURCED=Y 2024-09-10T19:04:01.559 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.559 INFO:tasks.workunit.client.0.smithi119.stderr:++ shopt -q login_shell 2024-09-10T19:04:01.559 INFO:tasks.workunit.client.0.smithi119.stderr:+++ umask 2024-09-10T19:04:01.559 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' 0022 -eq 0 ']' 2024-09-10T19:04:01.559 INFO:tasks.workunit.client.0.smithi119.stderr:++ SHELL=/bin/bash 2024-09-10T19:04:01.559 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.559 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-09-10T19:04:01.559 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.559 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/bash_completion.sh 2024-09-10T19:04:01.559 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.559 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-09-10T19:04:01.559 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.559 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-09-10T19:04:01.559 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.560 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/cobbler.sh 2024-09-10T19:04:01.560 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.560 INFO:tasks.workunit.client.0.smithi119.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-09-10T19:04:01.560 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.560 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-09-10T19:04:01.560 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.560 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorgrep.sh 2024-09-10T19:04:01.560 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.560 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.560 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.560 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.560 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.560 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.561 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.561 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.561 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.561 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.561 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.561 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.561 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.561 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.561 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.561 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.561 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.561 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.561 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.561 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.562 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.563 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.563 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.563 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.563 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.563 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_service 2024-09-10T19:04:01.563 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.563 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.563 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.563 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_service.log 2024-09-10T19:04:01.563 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.563 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.563 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.563 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.563 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.564 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/80087/exe 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/80060/exe 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_service: /' 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.565 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.566 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.567 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.567 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.567 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.567 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.567 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.567 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.567 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.567 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_cls_remote_reads 2024-09-10T19:04:01.567 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.567 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.567 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.567 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.567 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.567 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.568 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'grep=grep --color=auto' 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'egrep=egrep --color=auto' 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorls.sh 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' '!' -t 0 ']' 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:+++ return 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export S_COLORS=auto 2024-09-10T19:04:01.569 INFO:tasks.workunit.client.0.smithi119.stderr:+++ S_COLORS=auto 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_c_write_operations 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.570 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.571 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.571 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.571 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.571 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.571 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.571 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/80072/exe 2024-09-10T19:04:01.571 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.571 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-10T19:04:01.571 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.571 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-09-10T19:04:01.571 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-09-10T19:04:01.571 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-09-10T19:04:01.571 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.571 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:+++ /usr/libexec/grepconf.sh -c 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.572 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.573 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.574 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.574 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.574 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.574 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-09-10T19:04:01.574 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-09-10T19:04:01.574 INFO:tasks.workunit.client.0.smithi119.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-09-10T19:04:01.574 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.574 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.574 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-09-10T19:04:01.574 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.574 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/debuginfod.sh 2024-09-10T19:04:01.574 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-09-10T19:04:01.574 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.574 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-09-10T19:04:01.575 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.575 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/gawk.sh 2024-09-10T19:04:01.575 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.575 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-09-10T19:04:01.575 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.575 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/lang.sh 2024-09-10T19:04:01.575 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup 2024-09-10T19:04:01.575 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.575 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.575 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.575 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.575 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.575 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/locale 2024-09-10T19:04:01.575 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.576 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.576 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.576 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.576 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.576 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.576 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.576 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.576 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.576 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.576 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.576 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.577 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-09-10T19:04:01.577 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.577 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.577 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.577 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.577 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.577 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.577 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.577 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.577 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.577 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-09-10T19:04:01.577 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.577 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.578 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG_backup=en_US.UTF-8 2024-09-10T19:04:01.578 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.578 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /etc/locale.conf ']' 2024-09-10T19:04:01.578 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -x /usr/bin/sed ']' 2024-09-10T19:04:01.578 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/80124/exe 2024-09-10T19:04:01.578 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.578 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.578 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.578 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.578 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.578 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.578 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.579 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.579 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.579 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.579 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.579 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-10T19:04:01.579 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_api_c_read_operations 2024-09-10T19:04:01.579 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.579 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.579 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.579 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.579 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.579 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.579 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.579 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.580 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.580 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.580 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.580 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.580 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.580 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.580 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.580 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.580 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.580 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.580 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.580 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.581 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.581 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ list_parallel: /' 2024-09-10T19:04:01.581 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_list_parallel 2024-09-10T19:04:01.581 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/80185/exe 2024-09-10T19:04:01.581 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-10T19:04:01.581 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.581 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.581 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.581 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.581 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.581 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.581 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.582 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.582 INFO:tasks.workunit.client.0.smithi119.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-09-10T19:04:01.582 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.582 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.582 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.582 INFO:tasks.workunit.client.0.smithi119.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-09-10T19:04:01.582 INFO:tasks.workunit.client.0.smithi119.stderr:++++ export LANG=en_US.UTF-8 2024-09-10T19:04:01.582 INFO:tasks.workunit.client.0.smithi119.stderr:++++ LANG=en_US.UTF-8 2024-09-10T19:04:01.582 INFO:tasks.workunit.client.0.smithi119.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-09-10T19:04:01.582 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-09-10T19:04:01.582 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.582 INFO:tasks.workunit.client.0.smithi119.stderr:+++ LANG=en_US.UTF-8 2024-09-10T19:04:01.583 INFO:tasks.workunit.client.0.smithi119.stderr:+++ unset LANG_backup config 2024-09-10T19:04:01.583 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n '' ']' 2024-09-10T19:04:01.583 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-09-10T19:04:01.583 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' dumb = linux ']' 2024-09-10T19:04:01.583 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.583 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-09-10T19:04:01.583 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.583 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/less.sh 2024-09-10T19:04:01.583 INFO:tasks.workunit.client.0.smithi119.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-09-10T19:04:01.583 INFO:tasks.workunit.client.0.smithi119.stderr:++ for i in /etc/profile.d/*.sh 2024-09-10T19:04:01.583 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-09-10T19:04:01.583 INFO:tasks.workunit.client.0.smithi119.stderr:++ '[' '' ']' 2024-09-10T19:04:01.583 INFO:tasks.workunit.client.0.smithi119.stderr:++ . /etc/profile.d/which2.sh 2024-09-10T19:04:01.584 INFO:tasks.workunit.client.0.smithi119.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-09-10T19:04:01.584 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_delete_pools_parallel 2024-09-10T19:04:01.584 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-10T19:04:01.584 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-10T19:04:01.584 INFO:tasks.workunit.client.0.smithi119.stderr:+++++ readlink /proc/80155/exe 2024-09-10T19:04:01.584 INFO:tasks.workunit.client.0.smithi119.stderr:++++ basename /usr/bin/bash 2024-09-10T19:04:01.584 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_declare='declare -f' 2024-09-10T19:04:01.584 INFO:tasks.workunit.client.0.smithi119.stderr:+++ which_opt=-f 2024-09-10T19:04:01.584 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export which_declare 2024-09-10T19:04:01.584 INFO:tasks.workunit.client.0.smithi119.stderr:+++ export -f which 2024-09-10T19:04:01.584 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset i 2024-09-10T19:04:01.584 INFO:tasks.workunit.client.0.smithi119.stderr:++ unset -f pathmunge 2024-09-10T19:04:01.585 INFO:tasks.workunit.client.0.smithi119.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-09-10T19:04:01.585 INFO:tasks.workunit.client.0.smithi119.stderr:+ export PATH 2024-09-10T19:04:01.585 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-09-10T19:04:01.585 INFO:tasks.workunit.client.0.smithi119.stderr:+ unset rc 2024-09-10T19:04:01.585 INFO:tasks.workunit.client.0.smithi119.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-10T19:04:01.585 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph_test_rados_open_pools_parallel 2024-09-10T19:04:01.585 INFO:tasks.workunit.client.0.smithi119.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-10T19:04:02.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:02 smithi119 ceph-mon[29577]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:04:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3390683740' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-09-10T19:04:02.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:02 smithi119 ceph-mon[40674]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:04:02.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:02.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3390683740' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-09-10T19:04:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:02 smithi194 ceph-mon[32464]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:04:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3390683740' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-09-10T19:04:03.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:04:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:04:02] "GET /metrics HTTP/1.1" 200 225518 "" "Prometheus/2.43.0" 2024-09-10T19:04:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:03 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/368757669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi119-79512-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3096609987' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi119-79499-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2511332514' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi119-79477-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:03 smithi194 ceph-mon[32464]: osdmap e64: 8 total, 8 up, 8 in 2024-09-10T19:04:03.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:03 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:03.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/368757669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi119-79512-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:03.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3096609987' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi119-79499-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2511332514' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi119-79477-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:03 smithi119 ceph-mon[29577]: osdmap e64: 8 total, 8 up, 8 in 2024-09-10T19:04:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:03 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/368757669' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi119-79512-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3096609987' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi119-79499-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2511332514' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi119-79477-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:03 smithi119 ceph-mon[40674]: osdmap e64: 8 total, 8 up, 8 in 2024-09-10T19:04:05.687 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-10T19:04:05.687 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [----------] Global test environment set-up. 2024-09-10T19:04:05.687 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-10T19:04:05.687 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-10T19:04:05.687 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1 ms) 2024-09-10T19:04:05.687 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-10T19:04:05.687 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (0 ms) 2024-09-10T19:04:05.688 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-10T19:04:05.688 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-09-10T19:04:05.688 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-10T19:04:05.688 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (2 ms) 2024-09-10T19:04:05.688 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-10T19:04:05.688 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (1 ms) 2024-09-10T19:04:05.688 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-10T19:04:05.688 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (1 ms) 2024-09-10T19:04:05.688 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-10T19:04:05.688 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (0 ms) 2024-09-10T19:04:05.688 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-10T19:04:05.688 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (0 ms) 2024-09-10T19:04:05.689 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-10T19:04:05.689 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (0 ms) 2024-09-10T19:04:05.689 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-10T19:04:05.689 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1 ms) 2024-09-10T19:04:05.689 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-10T19:04:05.689 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (1 ms) 2024-09-10T19:04:05.689 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-10T19:04:05.689 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (1 ms) 2024-09-10T19:04:05.689 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [----------] 12 tests from AsioRados (10 ms total) 2024-09-10T19:04:05.689 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: 2024-09-10T19:04:05.689 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [----------] Global test environment tear-down 2024-09-10T19:04:05.689 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (4396 ms total) 2024-09-10T19:04:05.689 INFO:tasks.workunit.client.0.smithi119.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-10T19:04:07.455 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-09-10T19:04:07.455 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [----------] Global test environment set-up. 2024-09-10T19:04:07.456 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-10T19:04:07.456 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-10T19:04:07.456 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ OK ] LibRadosList.ListObjects (131 ms) 2024-09-10T19:04:07.456 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-10T19:04:07.456 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (18 ms) 2024-09-10T19:04:07.456 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-10T19:04:07.456 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: myset foo1,foo2,foo3 2024-09-10T19:04:07.456 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo1 2024-09-10T19:04:07.456 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo2 2024-09-10T19:04:07.456 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo3 2024-09-10T19:04:07.456 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: myset foo1,foo4,foo5 2024-09-10T19:04:07.456 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo4 2024-09-10T19:04:07.457 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo5 2024-09-10T19:04:07.457 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo1 2024-09-10T19:04:07.457 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: myset foo6,foo7 2024-09-10T19:04:07.457 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo7 2024-09-10T19:04:07.457 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo6 2024-09-10T19:04:07.457 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-10T19:04:07.457 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: ns1:foo4 2024-09-10T19:04:07.457 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: ns1:foo5 2024-09-10T19:04:07.457 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: ns2:foo7 2024-09-10T19:04:07.457 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: ns2:foo6 2024-09-10T19:04:07.457 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: ns1:foo1 2024-09-10T19:04:07.457 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: :foo1 2024-09-10T19:04:07.457 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: :foo2 2024-09-10T19:04:07.457 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: :foo3 2024-09-10T19:04:07.458 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (52 ms) 2024-09-10T19:04:07.458 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-10T19:04:07.458 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 1 0 2024-09-10T19:04:07.458 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 10 0 2024-09-10T19:04:07.458 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 13 0 2024-09-10T19:04:07.458 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 7 0 2024-09-10T19:04:07.458 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 14 0 2024-09-10T19:04:07.458 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 0 0 2024-09-10T19:04:07.458 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 15 0 2024-09-10T19:04:07.458 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 11 0 2024-09-10T19:04:07.458 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 5 0 2024-09-10T19:04:07.458 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 8 0 2024-09-10T19:04:07.458 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 6 0 2024-09-10T19:04:07.458 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 3 0 2024-09-10T19:04:07.459 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 4 0 2024-09-10T19:04:07.459 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 12 0 2024-09-10T19:04:07.459 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 9 0 2024-09-10T19:04:07.459 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 2 0 2024-09-10T19:04:07.459 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-10T19:04:07.459 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (51 ms) 2024-09-10T19:04:07.459 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-10T19:04:07.459 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: x cursor=MIN 2024-09-10T19:04:07.459 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=1 cursor=14:02547ec2:::1:head 2024-09-10T19:04:07.459 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=10 cursor=14:52ea6a34:::10:head 2024-09-10T19:04:07.459 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=13 cursor=14:566253c9:::13:head 2024-09-10T19:04:07.459 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=7 cursor=14:5c6b0b28:::7:head 2024-09-10T19:04:07.459 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=14 cursor=14:62a1935d:::14:head 2024-09-10T19:04:07.459 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=0 cursor=14:6cac518f:::0:head 2024-09-10T19:04:07.460 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=15 cursor=14:863748b0:::15:head 2024-09-10T19:04:07.460 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=11 cursor=14:89d3ae78:::11:head 2024-09-10T19:04:07.460 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=5 cursor=14:b29083e3:::5:head 2024-09-10T19:04:07.460 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=8 cursor=14:bd63b0f1:::8:head 2024-09-10T19:04:07.460 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=6 cursor=14:c4fdafeb:::6:head 2024-09-10T19:04:07.460 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=3 cursor=14:cfc208b3:::3:head 2024-09-10T19:04:07.460 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=4 cursor=14:d83876eb:::4:head 2024-09-10T19:04:07.460 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=12 cursor=14:de5d7c5f:::12:head 2024-09-10T19:04:07.460 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=9 cursor=14:e960b815:::9:head 2024-09-10T19:04:07.460 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > oid=2 cursor=14:f905c69b:::2:head 2024-09-10T19:04:07.460 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-10T19:04:07.460 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=1 cursor=14:02547ec2:::1:head 2024-09-10T19:04:07.460 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:02547ec2:::1:head 2024-09-10T19:04:07.461 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:02547ec2:::1:head -> 1 2024-09-10T19:04:07.461 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=10 cursor=14:52ea6a34:::10:head 2024-09-10T19:04:07.461 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:52ea6a34:::10:head 2024-09-10T19:04:07.461 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:52ea6a34:::10:head -> 10 2024-09-10T19:04:07.461 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=13 cursor=14:566253c9:::13:head 2024-09-10T19:04:07.461 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:566253c9:::13:head 2024-09-10T19:04:07.461 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:566253c9:::13:head -> 13 2024-09-10T19:04:07.461 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=7 cursor=14:5c6b0b28:::7:head 2024-09-10T19:04:07.461 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:5c6b0b28:::7:head 2024-09-10T19:04:07.461 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:5c6b0b28:::7:head -> 7 2024-09-10T19:04:07.509 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=14 cursor=14:62a1935d:::14:head 2024-09-10T19:04:07.509 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:62a1935d:::14:head 2024-09-10T19:04:07.509 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:62a1935d:::14:head -> 14 2024-09-10T19:04:07.509 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=0 cursor=14:6cac518f:::0:head 2024-09-10T19:04:07.509 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:6cac518f:::0:head 2024-09-10T19:04:07.509 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:6cac518f:::0:head -> 0 2024-09-10T19:04:07.509 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=15 cursor=14:863748b0:::15:head 2024-09-10T19:04:07.509 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:863748b0:::15:head 2024-09-10T19:04:07.510 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:863748b0:::15:head -> 15 2024-09-10T19:04:07.510 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=11 cursor=14:89d3ae78:::11:head 2024-09-10T19:04:07.510 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:89d3ae78:::11:head 2024-09-10T19:04:07.510 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:89d3ae78:::11:head -> 11 2024-09-10T19:04:07.510 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=5 cursor=14:b29083e3:::5:head 2024-09-10T19:04:07.510 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:b29083e3:::5:head 2024-09-10T19:04:07.510 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:b29083e3:::5:head -> 5 2024-09-10T19:04:07.510 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=8 cursor=14:bd63b0f1:::8:head 2024-09-10T19:04:07.510 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:bd63b0f1:::8:head 2024-09-10T19:04:07.510 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:bd63b0f1:::8:head -> 8 2024-09-10T19:04:07.510 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=6 cursor=14:c4fdafeb:::6:head 2024-09-10T19:04:07.510 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:c4fdafeb:::6:head 2024-09-10T19:04:07.510 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:c4fdafeb:::6:head -> 6 2024-09-10T19:04:07.511 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=3 cursor=14:cfc208b3:::3:head 2024-09-10T19:04:07.511 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:cfc208b3:::3:head 2024-09-10T19:04:07.511 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:cfc208b3:::3:head -> 3 2024-09-10T19:04:07.511 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=4 cursor=14:d83876eb:::4:head 2024-09-10T19:04:07.511 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:d83876eb:::4:head 2024-09-10T19:04:07.511 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:d83876eb:::4:head -> 4 2024-09-10T19:04:07.511 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=12 cursor=14:de5d7c5f:::12:head 2024-09-10T19:04:07.511 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:de5d7c5f:::12:head 2024-09-10T19:04:07.511 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:de5d7c5f:::12:head -> 12 2024-09-10T19:04:07.511 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=9 cursor=14:e960b815:::9:head 2024-09-10T19:04:07.511 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:e960b815:::9:head 2024-09-10T19:04:07.511 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:e960b815:::9:head -> 9 2024-09-10T19:04:07.511 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : oid=2 cursor=14:f905c69b:::2:head 2024-09-10T19:04:07.511 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:f905c69b:::2:head 2024-09-10T19:04:07.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:f905c69b:::2:head -> 2 2024-09-10T19:04:07.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:5c6b0b28:::7:head 2024-09-10T19:04:07.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:5c6b0b28:::7:head expected=14:5c6b0b28:::7:head 2024-09-10T19:04:07.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:5c6b0b28:::7:head -> 7 2024-09-10T19:04:07.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=7 expected=7 2024-09-10T19:04:07.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:f905c69b:::2:head 2024-09-10T19:04:07.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:f905c69b:::2:head expected=14:f905c69b:::2:head 2024-09-10T19:04:07.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:f905c69b:::2:head -> 2 2024-09-10T19:04:07.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=2 expected=2 2024-09-10T19:04:07.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:e960b815:::9:head 2024-09-10T19:04:07.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:e960b815:::9:head expected=14:e960b815:::9:head 2024-09-10T19:04:07.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:e960b815:::9:head -> 9 2024-09-10T19:04:07.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=9 expected=9 2024-09-10T19:04:07.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:b29083e3:::5:head 2024-09-10T19:04:07.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:b29083e3:::5:head expected=14:b29083e3:::5:head 2024-09-10T19:04:07.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:b29083e3:::5:head -> 5 2024-09-10T19:04:07.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=5 expected=5 2024-09-10T19:04:07.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:89d3ae78:::11:head 2024-09-10T19:04:07.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:89d3ae78:::11:head expected=14:89d3ae78:::11:head 2024-09-10T19:04:07.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:89d3ae78:::11:head -> 11 2024-09-10T19:04:07.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=11 expected=11 2024-09-10T19:04:07.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:863748b0:::15:head 2024-09-10T19:04:07.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:863748b0:::15:head expected=14:863748b0:::15:head 2024-09-10T19:04:07.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:863748b0:::15:head -> 15 2024-09-10T19:04:07.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=15 expected=15 2024-09-10T19:04:07.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:02547ec2:::1:head 2024-09-10T19:04:07.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:02547ec2:::1:head expected=14:02547ec2:::1:head 2024-09-10T19:04:07.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:02547ec2:::1:head -> 1 2024-09-10T19:04:07.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=1 expected=1 2024-09-10T19:04:07.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:6cac518f:::0:head 2024-09-10T19:04:07.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:6cac518f:::0:head expected=14:6cac518f:::0:head 2024-09-10T19:04:07.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:6cac518f:::0:head -> 0 2024-09-10T19:04:07.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: pgmap v59: 260 pgs: 128 creating+peering, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:04:07.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:07.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3073983891' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi119-79536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/150333045' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi119-79573-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4156622590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi119-79837-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3801067030' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi119-79610-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi119-79624-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: pgmap v61: 420 pgs: 160 unknown, 128 creating+peering, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: osdmap e65: 8 total, 8 up, 8 in 2024-09-10T19:04:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: Health check failed: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi119-79512-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi119-79499-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi119-79477-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi119-79536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi119-79573-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi119-79837-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi119-79610-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi119-79624-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:07.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: pgmap v59: 260 pgs: 128 creating+peering, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:04:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3073983891' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi119-79536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/150333045' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi119-79573-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4156622590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi119-79837-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3801067030' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi119-79610-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi119-79624-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: pgmap v61: 420 pgs: 160 unknown, 128 creating+peering, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: osdmap e65: 8 total, 8 up, 8 in 2024-09-10T19:04:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: Health check failed: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi119-79512-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi119-79499-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi119-79477-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi119-79536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi119-79573-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi119-79837-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi119-79610-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi119-79624-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:07.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: pgmap v59: 260 pgs: 128 creating+peering, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:04:07.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:07.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3073983891' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi119-79536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/150333045' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi119-79573-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4156622590' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi119-79837-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3801067030' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi119-79610-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi119-79624-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: pgmap v61: 420 pgs: 160 unknown, 128 creating+peering, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:04:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: osdmap e65: 8 total, 8 up, 8 in 2024-09-10T19:04:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: Health check failed: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi119-79512-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi119-79499-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi119-79477-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi119-79536-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi119-79573-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi119-79837-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi119-79610-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi119-79624-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:07.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:08.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:04:08.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2699128120' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1923597735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi119-80015-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/342381034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi119-79782-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2911793265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi119-79823-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4022463113' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi119-79960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: pgmap v63: 708 pgs: 448 unknown, 128 creating+peering, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:04:08.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: osdmap e66: 8 total, 8 up, 8 in 2024-09-10T19:04:08.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:08.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi119-79782-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi119-80015-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi119-79823-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2440869202' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80075-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi119-79960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2705807411' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80038-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:08.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi119-79512-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi119-79499-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi119-79477-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi119-79536-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi119-79573-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi119-79837-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi119-79610-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi119-79624-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/487275958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/922104191' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1046337846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi119-80244-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4231703072' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi119-79868-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: osdmap e67: 8 total, 8 up, 8 in 2024-09-10T19:04:08.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi119-80244-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1036072373' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T19:04:08.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi119-79868-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:08.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2699128120' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1923597735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi119-80015-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/342381034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi119-79782-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2911793265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi119-79823-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4022463113' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi119-79960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: pgmap v63: 708 pgs: 448 unknown, 128 creating+peering, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:04:08.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: osdmap e66: 8 total, 8 up, 8 in 2024-09-10T19:04:08.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:08.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi119-79782-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi119-80015-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi119-79823-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2440869202' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80075-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi119-79960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2705807411' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80038-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2699128120' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1923597735' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi119-80015-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/342381034' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi119-79782-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2911793265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi119-79823-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4022463113' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi119-79960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: pgmap v63: 708 pgs: 448 unknown, 128 creating+peering, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:04:08.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: osdmap e66: 8 total, 8 up, 8 in 2024-09-10T19:04:08.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:08.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi119-79782-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi119-80015-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi119-79823-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2440869202' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80075-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi119-79960-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2705807411' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80038-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:08.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi119-79512-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi119-79499-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi119-79477-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi119-79536-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi119-79573-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi119-79837-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi119-79610-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi119-79624-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/487275958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/922104191' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1046337846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi119-80244-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4231703072' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi119-79868-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: osdmap e67: 8 total, 8 up, 8 in 2024-09-10T19:04:08.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.572 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi119-80244-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.572 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1036072373' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T19:04:08.572 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi119-79868-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.572 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:08.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:08.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi119-79512-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi119-79499-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi119-79477-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi119-79536-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi119-79573-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi119-79837-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi119-79610-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi119-79624-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:08.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/487275958' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/922104191' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1046337846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi119-80244-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4231703072' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi119-79868-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: osdmap e67: 8 total, 8 up, 8 in 2024-09-10T19:04:08.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80217-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi119-80244-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1036072373' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T19:04:08.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi119-79868-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:08.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:09.182 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=0 list_parallel: process_1_[80278]: starting. 2024-09-10T19:04:09.182 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_1_[80278]: creating pool ceph_test_rados_list_parallel.smithi119-80260 2024-09-10T19:04:09.182 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_1_[80278]: created object 0... 2024-09-10T19:04:09.182 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_1_[80278]: created object 25... 2024-09-10T19:04:09.182 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_1_[80278]: created object 49... 2024-09-10T19:04:09.183 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_1_[80278]: finishing. 2024-09-10T19:04:09.183 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_1_[80278]: shutting down. 2024-09-10T19:04:09.183 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_2_[80279]: starting. 2024-09-10T19:04:09.183 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_2_[80279]: listing objects. 2024-09-10T19:04:09.183 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_2_[80279]: listed object 0... 2024-09-10T19:04:09.183 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_2_[80279]: listed object 25... 2024-09-10T19:04:09.183 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_2_[80279]: saw 50 objects 2024-09-10T19:04:09.183 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_2_[80279]: shutting down. 2024-09-10T19:04:09.183 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.183 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_3_[81018]: starting. 2024-09-10T19:04:09.183 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_3_[81018]: creating pool ceph_test_rados_list_parallel.smithi119-80260 2024-09-10T19:04:09.183 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_3_[81018]: created object 0... 2024-09-10T19:04:09.183 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_3_[81018]: created object 25... 2024-09-10T19:04:09.184 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_3_[81018]: created object 49... 2024-09-10T19:04:09.184 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_3_[81018]: finishing. 2024-09-10T19:04:09.184 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_3_[81018]: shutting down. 2024-09-10T19:04:09.184 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.184 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_4_[81019]: starting. 2024-09-10T19:04:09.184 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_4_[81019]: listing objects. 2024-09-10T19:04:09.184 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_4_[81019]: listed object 0... 2024-09-10T19:04:09.184 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_4_[81019]: listed object 25... 2024-09-10T19:04:09.184 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_4_[81019]: saw 42 objects 2024-09-10T19:04:09.184 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_4_[81019]: shutting down. 2024-09-10T19:04:09.184 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.184 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_5_[81020]: starting. 2024-09-10T19:04:09.184 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_5_[81020]: removed 25 objects... 2024-09-10T19:04:09.185 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_5_[81020]: removed half of the objects 2024-09-10T19:04:09.185 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_5_[81020]: removed 50 objects... 2024-09-10T19:04:09.185 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_5_[81020]: removed 50 objects 2024-09-10T19:04:09.185 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_5_[81020]: shutting down. 2024-09-10T19:04:09.185 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.185 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.185 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_6_[81076]: starting. 2024-09-10T19:04:09.185 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_6_[81076]: creating pool ceph_test_rados_list_parallel.smithi119-80260 2024-09-10T19:04:09.185 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_6_[81076]: created object 0... 2024-09-10T19:04:09.185 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_6_[81076]: created object 25... 2024-09-10T19:04:09.185 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_6_[81076]: created object 49... 2024-09-10T19:04:09.185 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_6_[81076]: finishing. 2024-09-10T19:04:09.185 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_6_[81076]: shutting down. 2024-09-10T19:04:09.185 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_7_[81077]: starting. 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_7_[81077]: listing objects. 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_7_[81077]: listed object 0... 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_7_[81077]: listed object 25... 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_7_[81077]: saw 50 objects 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_7_[81077]: shutting down. 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_8_[81078]: starting. 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_8_[81078]: added 25 objects... 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_8_[81078]: added half of the objects 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_8_[81078]: added 50 objects... 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_8_[81078]: added 50 objects 2024-09-10T19:04:09.186 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_8_[81078]: shutting down. 2024-09-10T19:04:09.187 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.187 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.187 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.187 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_9_[81128]: starting. 2024-09-10T19:04:09.642 INFO:tasks.workunit.client.0.smithi119.stdout: api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-09-10T19:04:09.642 INFO:tasks.workunit.client.0.smithi119.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-09-10T19:04:09.642 INFO:tasks.workunit.client.0.smithi119.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-09-10T19:04:09.642 INFO:tasks.workunit.client.0.smithi119.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-09-10T19:04:09.642 INFO:tasks.workunit.client.0.smithi119.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (8138 ms) 2024-09-10T19:04:09.642 INFO:tasks.workunit.client.0.smithi119.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (8138 ms total) 2024-09-10T19:04:09.642 INFO:tasks.workunit.client.0.smithi119.stdout: api_cls_remote_reads: 2024-09-10T19:04:09.642 INFO:tasks.workunit.client.0.smithi119.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-09-10T19:04:09.642 INFO:tasks.workunit.client.0.smithi119.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (8138 ms total) 2024-09-10T19:04:09.643 INFO:tasks.workunit.client.0.smithi119.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-09-10T19:04:09.657 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-09-10T19:04:09.658 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-10T19:04:09.658 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-10T19:04:09.658 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-10T19:04:09.658 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-10T19:04:09.658 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (78 ms) 2024-09-10T19:04:09.658 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-10T19:04:09.658 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (40 ms) 2024-09-10T19:04:09.658 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-10T19:04:09.658 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (8105 ms) 2024-09-10T19:04:09.658 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (8223 ms total) 2024-09-10T19:04:09.658 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: 2024-09-10T19:04:09.658 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-10T19:04:09.658 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (8224 ms total) 2024-09-10T19:04:09.659 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-10T19:04:09.680 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-10T19:04:09.680 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-10T19:04:09.680 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-10T19:04:09.680 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-10T19:04:09.680 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-10T19:04:09.680 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (58 ms) 2024-09-10T19:04:09.681 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-10T19:04:09.681 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (69 ms) 2024-09-10T19:04:09.681 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-10T19:04:09.681 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (8103 ms) 2024-09-10T19:04:09.681 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-10T19:04:09.681 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.117028+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:09.681 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.117122+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi119-79782-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:09.681 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.117217+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi119-80015-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:09.681 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.117284+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi119-79823-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:09.681 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.117341+0000 mon.a [INF] from='client.? 172.21.15.119:0/2440869202' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80075-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:09.681 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.117383+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:09.681 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.117437+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi119-79960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:09.681 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.117494+0000 mon.a [INF] from='client.? 172.21.15.119:0/2705807411' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80038-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:09.682 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.117668+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:09.682 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.117764+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80217-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:09.682 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.117835+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:09.682 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.117905+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi119-80244-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:09.937 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.117998+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd list_parallel: process_9_[81128]: creating pool ceph_test_rados_list_parallel.smithi119-80260 2024-09-10T19:04:09.938 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_9_[81128]: created object 0... 2024-09-10T19:04:09.938 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_9_[81128]: created object 25... 2024-09-10T19:04:09.938 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_9_[81128]: created object 49... 2024-09-10T19:04:09.938 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_9_[81128]: finishing. 2024-09-10T19:04:09.938 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_9_[81128]: shutting down. 2024-09-10T19:04:09.938 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.938 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.938 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.938 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_10_[81129]: starting. 2024-09-10T19:04:09.938 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_10_[81129]: listing objects. 2024-09-10T19:04:09.939 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_10_[81129]: listed object 0... 2024-09-10T19:04:09.939 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_10_[81129]: listed object 25... 2024-09-10T19:04:09.939 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_10_[81129]: listed object 50... 2024-09-10T19:04:09.939 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_10_[81129]: listed object 75... 2024-09-10T19:04:09.939 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_10_[81129]: saw 92 objects 2024-09-10T19:04:09.939 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_10_[81129]: shutting down. 2024-09-10T19:04:09.939 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.939 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.939 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.939 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_13_[81132]: starting. 2024-09-10T19:04:09.939 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_13_[81132]: removed 25 objects... 2024-09-10T19:04:09.939 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_13_[81132]: removed half of the objects 2024-09-10T19:04:09.939 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_13_[81132]: removed 50 objects... 2024-09-10T19:04:09.939 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_13_[81132]: removed 50 objects 2024-09-10T19:04:09.940 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_13_[81132]: shutting down. 2024-09-10T19:04:09.940 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.940 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.940 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.940 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_11_[81130]: starting. 2024-09-10T19:04:09.940 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_11_[81130]: added 25 objects... 2024-09-10T19:04:09.940 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_11_[81130]: added half of the objects 2024-09-10T19:04:09.940 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_11_[81130]: added 50 objects... 2024-09-10T19:04:09.940 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_11_[81130]: added 50 objects 2024-09-10T19:04:09.940 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_11_[81130]: shutting down. 2024-09-10T19:04:09.940 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.940 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.940 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.941 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_12_[81131]: starting. 2024-09-10T19:04:09.941 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_12_[81131]: added 25 objects... 2024-09-10T19:04:09.941 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_12_[81131]: added half of the objects 2024-09-10T19:04:09.941 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_12_[81131]: added 50 objects... 2024-09-10T19:04:09.941 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_12_[81131]: added 50 objects 2024-09-10T19:04:09.941 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_12_[81131]: shutting down. 2024-09-10T19:04:09.941 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.941 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.941 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.941 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.941 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_14_[81272]: starting. 2024-09-10T19:04:09.941 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_14_[81272]: creating pool ceph_test_rados_list_parallel.smithi119-80260 2024-09-10T19:04:09.941 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_14_[81272]: created object 0... 2024-09-10T19:04:09.942 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_14_[81272]: created object 25... 2024-09-10T19:04:09.942 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_14_[81272]: created object 49... 2024-09-10T19:04:09.942 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_14_[81272]: finishing. 2024-09-10T19:04:09.942 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_14_[81272]: shutting down. 2024-09-10T19:04:09.942 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.942 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.942 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.942 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:09.942 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_16_[81274]: starting. 2024-09-10T19:04:09.942 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_16_[81274]: added 25 objects... 2024-09-10T19:04:09.942 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_16_[81274]: added half of the objects 2024-09-10T19:04:09.942 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_16_[81274]: added 50 objects... 2024-09-10T19:04:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi119-79782-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi119-80015-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi119-79823-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2440869202' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80075-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi119-79960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2705807411' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80038-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:10.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80217-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi119-80244-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi119-79868-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: osdmap e68: 8 total, 8 up, 8 in 2024-09-10T19:04:10.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi119-80068-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:10.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi119-80068-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:10.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:10.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:10.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:10.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:10.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:10.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi119-79782-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi119-80015-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi119-79823-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2440869202' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80075-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi119-79960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2705807411' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80038-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80217-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi119-80244-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi119-79868-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: osdmap e68: 8 total, 8 up, 8 in 2024-09-10T19:04:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi119-80068-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi119-80068-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:10.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:10.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi119-79782-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi119-80015-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi119-79823-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2440869202' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80075-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi119-79960-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2705807411' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80038-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:10.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80217-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi119-80244-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi119-79868-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: osdmap e68: 8 total, 8 up, 8 in 2024-09-10T19:04:10.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi119-80068-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:10.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi119-80068-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:10.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:10.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:10.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:10.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:10.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:10.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:10.372 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_16_[81274]: added 50 objects 2024-09-10T19:04:10.373 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_16_[81274]: shutting down. 2024-09-10T19:04:10.373 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:10.373 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:10.373 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:10.373 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:10.373 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_15_[81273]: starting. 2024-09-10T19:04:10.373 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_15_[81273]: listing objects. 2024-09-10T19:04:10.373 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_15_[81273]: listed object 0... 2024-09-10T19:04:10.373 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_15_[81273]: listed object 25... 2024-09-10T19:04:10.373 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_15_[81273]: listed object 50... 2024-09-10T19:04:10.373 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_15_[81273]: listed object 75... 2024-09-10T19:04:10.374 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_15_[81273]: listed object 100... 2024-09-10T19:04:10.374 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_15_[81273]: listed object 125... 2024-09-10T19:04:10.374 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_15_[81273]: saw 150 objects 2024-09-10T19:04:10.374 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: process_15_[81273]: shutting down. 2024-09-10T19:04:10.374 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:10.374 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:10.374 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:10.374 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:10.374 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******************************* 2024-09-10T19:04:10.374 INFO:tasks.workunit.client.0.smithi119.stdout: list_parallel: ******* SUCCESS ********** 2024-09-10T19:04:10.378 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-10T19:04:10.378 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [==========] Running 16 tests from 1 test suite. 2024-09-10T19:04:10.378 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-10T19:04:10.378 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest 2024-09-10T19:04:10.378 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-10T19:04:10.379 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-10T19:04:10.379 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-10T19:04:10.379 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (132 ms) 2024-09-10T19:04:10.379 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-10T19:04:10.379 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (66 ms) 2024-09-10T19:04:10.379 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-10T19:04:10.379 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (23 ms) 2024-09-10T19:04:10.379 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-10T19:04:10.379 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (23 ms) 2024-09-10T19:04:10.379 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-10T19:04:10.379 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (6 ms) 2024-09-10T19:04:10.380 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-10T19:04:10.380 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (8 ms) 2024-09-10T19:04:10.380 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-10T19:04:10.380 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (4 ms) 2024-09-10T19:04:10.380 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-10T19:04:10.380 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (7 ms) 2024-09-10T19:04:10.380 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-10T19:04:10.380 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (2 ms) 2024-09-10T19:04:10.380 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-10T19:04:10.380 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (5 ms) 2024-09-10T19:04:10.380 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-10T19:04:10.380 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (4 ms) 2024-09-10T19:04:10.380 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-10T19:04:10.380 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (13 ms) 2024-09-10T19:04:10.381 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-10T19:04:10.381 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (10 ms) 2024-09-10T19:04:10.381 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-10T19:04:10.381 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (6 ms) 2024-09-10T19:04:10.381 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-10T19:04:10.381 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (4 ms) 2024-09-10T19:04:10.381 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [----------] 16 tests from CReadOpsTest (314 ms total) 2024-09-10T19:04:10.381 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: 2024-09-10T19:04:10.381 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-10T19:04:10.381 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [==========] 16 tests from 1 test suite ran. (8872 ms total) 2024-09-10T19:04:10.381 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_read_operations: [ PASSED ] 16 tests. 2024-09-10T19:04:10.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:04:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:04:10.904 INFO:tasks.workunit.client.0.smithi119.stdout:pool application enable","pool": "LibRadosStat_smithi119-79868-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.904 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.120886+0000 mon.b [INF] from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi119-80068-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:10.904 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.126731+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi119-80068-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:10.904 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.192089+0000 mon.c [INF] from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:10.904 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:08.474528+0000 mon.b [INF] from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:10.904 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.027144+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:10.904 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.027792+0000 mon.c [INF] from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:10.904 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.635694+0000 mon.c [INF] from='client.? 172.21.15.119:0/1013785956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:10.904 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.636428+0000 mon.c [INF] from='client.? 172.21.15.119:0/3064027944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi119-79477-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:10.904 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.637957+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:10.905 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.638429+0000 mon.c [INF] from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:10.905 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.645617+0000 mon.c [INF] from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi119-79512-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:10.905 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.657082+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:10.905 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.661599+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi119-79477-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:10.905 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.662198+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:10.905 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.662722+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi119-79512-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:10.905 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.679468+0000 mon.c [INF] from='client.? 172.21.15.119:0/4244195218' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-10T19:04:10.905 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.679649+0000 client.admin [INF] onexx 2024-09-10T19:04:10.905 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.945844+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-10T19:04:10.905 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.946050+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:10.905 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:09.947010+0000 mon.b [INF] from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:10.905 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:10.211999+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:10.905 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:10.213123+0000 mon.b [INF] from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:10.906 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:10.214074+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:10.906 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:10.366813+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi119-80068-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]': finished 2024-09-10T19:04:10.906 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:10.367101+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.906 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:10.367191+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi119-79477-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.906 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:10.367302+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:10.906 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:10.367429+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi119-79512-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:10.906 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:10.367712+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:10.906 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:10.371956+0000 mon.c [INF] from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi119-79512-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:11.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: pgmap v66: 804 pgs: 32 unknown, 480 creating+peering, 292 active+clean; 584 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 12 KiB/s wr, 23 op/s 2024-09-10T19:04:11.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3852591916' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T19:04:11.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:11.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: osdmap e69: 8 total, 8 up, 8 in 2024-09-10T19:04:11.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1013785956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3064027944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi119-79477-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:11.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi119-79512-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:11.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi119-79477-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi119-79512-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:11.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4244195218' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-10T19:04:11.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: onexx 2024-09-10T19:04:11.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-10T19:04:11.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:11.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:11.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:11.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:11.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:11.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi119-80068-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]': finished 2024-09-10T19:04:11.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:11.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi119-79477-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:11.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:11.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi119-79512-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:11.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:11.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: osdmap e70: 8 total, 8 up, 8 in 2024-09-10T19:04:11.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi119-79512-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:11.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi119-79512-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:11.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1548473117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: pgmap v66: 804 pgs: 32 unknown, 480 creating+peering, 292 active+clean; 584 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 12 KiB/s wr, 23 op/s 2024-09-10T19:04:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3852591916' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T19:04:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: osdmap e69: 8 total, 8 up, 8 in 2024-09-10T19:04:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1013785956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3064027944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi119-79477-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi119-79512-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi119-79477-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi119-79512-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4244195218' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: onexx 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi119-80068-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]': finished 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi119-79477-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:11.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi119-79512-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:11.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:11.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: osdmap e70: 8 total, 8 up, 8 in 2024-09-10T19:04:11.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi119-79512-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:11.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi119-79512-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:11.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1548473117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: pgmap v66: 804 pgs: 32 unknown, 480 creating+peering, 292 active+clean; 584 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 12 KiB/s wr, 23 op/s 2024-09-10T19:04:11.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3852591916' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T19:04:11.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:11.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: osdmap e69: 8 total, 8 up, 8 in 2024-09-10T19:04:11.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1013785956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3064027944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi119-79477-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:11.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi119-79512-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:11.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi119-79477-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi119-79512-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:11.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4244195218' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-10T19:04:11.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: onexx 2024-09-10T19:04:11.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-09-10T19:04:11.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:11.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:11.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:11.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:11.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:11.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi119-80068-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]': finished 2024-09-10T19:04:11.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:11.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi119-79477-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:11.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:11.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi119-79512-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:11.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:11.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: osdmap e70: 8 total, 8 up, 8 in 2024-09-10T19:04:11.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi119-79512-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:11.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi119-79512-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:11.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1548473117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:11.474 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:10.373567+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi119-79512-16", api_service: [==========] Running 4 tests from 1 test suite. 2024-09-10T19:04:11.475 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: [----------] Global test environment set-up. 2024-09-10T19:04:11.475 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-10T19:04:11.475 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-10T19:04:11.475 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5051 ms) 2024-09-10T19:04:11.475 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-10T19:04:11.475 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: [ OK ] LibRadosService.RegisterLate (14 ms) 2024-09-10T19:04:11.475 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-10T19:04:11.475 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: cluster: 2024-09-10T19:04:11.475 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: id: 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T19:04:11.475 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: health: HEALTH_WARN 2024-09-10T19:04:11.475 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 9 pool(s) do not have an application enabled 2024-09-10T19:04:11.475 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:11.476 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: services: 2024-09-10T19:04:11.476 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-09-10T19:04:11.476 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: mgr: y(active, since 110s), standbys: x 2024-09-10T19:04:11.476 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 3m) 2024-09-10T19:04:11.476 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-10T19:04:11.476 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-09-10T19:04:11.476 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:11.476 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: data: 2024-09-10T19:04:11.476 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: pools: 15 pools, 420 pgs 2024-09-10T19:04:11.476 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: objects: 196 objects, 582 KiB 2024-09-10T19:04:11.476 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:11.476 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: pgs: 38.095% pgs unknown 2024-09-10T19:04:11.476 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 30.476% pgs not active 2024-09-10T19:04:11.477 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 160 unknown 2024-09-10T19:04:11.477 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 132 active+clean 2024-09-10T19:04:11.477 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 128 creating+peering 2024-09-10T19:04:11.477 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:11.477 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: io: 2024-09-10T19:04:11.477 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-09-10T19:04:11.477 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:11.477 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:11.477 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: cluster: 2024-09-10T19:04:11.477 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: id: 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T19:04:11.477 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: health: HEALTH_WARN 2024-09-10T19:04:11.477 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 18 pool(s) do not have an application enabled 2024-09-10T19:04:11.477 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: services: 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: mgr: y(active, since 112s), standbys: x 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 3m) 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: foo: 8 portals active (1 hosts, 3 zones) 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: data: 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: pools: 24 pools, 708 pgs 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: objects: 196 objects, 582 KiB 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: pgs: 63.277% pgs unknown 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 18.079% pgs not active 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 448 unknown 2024-09-10T19:04:11.478 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 132 active+clean 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 128 creating+peering 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: io: 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: cluster: 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: id: 3afe9804-6fa6-11ef-bcea-c7b262605968 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: health: HEALTH_WARN 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 5 pool(s) do not have an application enabled 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: services: 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: mgr: y(active, since 114s), standbys: x 2024-09-10T19:04:11.479 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 3m) 2024-09-10T19:04:12.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi119-79536-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:12.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2777964774' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:12.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:12.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:12.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.0"}]: dispatch 2024-09-10T19:04:12.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.0"}]: dispatch 2024-09-10T19:04:12.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.1"}]: dispatch 2024-09-10T19:04:12.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.1"}]: dispatch 2024-09-10T19:04:12.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.2"}]: dispatch 2024-09-10T19:04:12.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.2"}]: dispatch 2024-09-10T19:04:12.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.3"}]: dispatch 2024-09-10T19:04:12.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.3"}]: dispatch 2024-09-10T19:04:12.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.4"}]: dispatch 2024-09-10T19:04:12.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.4"}]: dispatch 2024-09-10T19:04:12.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.5"}]: dispatch 2024-09-10T19:04:12.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.5"}]: dispatch 2024-09-10T19:04:12.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.6"}]: dispatch 2024-09-10T19:04:12.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.6"}]: dispatch 2024-09-10T19:04:12.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.7"}]: dispatch 2024-09-10T19:04:12.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.7"}]: dispatch 2024-09-10T19:04:12.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.8"}]: dispatch 2024-09-10T19:04:12.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.8"}]: dispatch 2024-09-10T19:04:12.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.9"}]: dispatch 2024-09-10T19:04:12.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.9"}]: dispatch 2024-09-10T19:04:12.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-09-10T19:04:12.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4244195218' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-09-10T19:04:12.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: twoxx 2024-09-10T19:04:12.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi119-79536-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:12.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:12.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:12.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: pgmap v69: 588 pgs: 232 unknown, 128 creating+peering, 228 active+clean; 584 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 16 KiB/s wr, 29 op/s 2024-09-10T19:04:12.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:12.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:12.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:12.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:12.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi119-79868-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:12.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:12.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:12.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:12.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1126701021' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T19:04:12.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:12.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1548473117' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:12.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1013785956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:12.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi119-79868-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:12.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:12.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: osdmap e71: 8 total, 8 up, 8 in 2024-09-10T19:04:12.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi119-79868-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:12.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi119-80015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:12.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-09-10T19:04:12.273 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:12.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi119-79536-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:12.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2777964774' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:12.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:12.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:12.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.0"}]: dispatch 2024-09-10T19:04:12.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.0"}]: dispatch 2024-09-10T19:04:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.1"}]: dispatch 2024-09-10T19:04:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.1"}]: dispatch 2024-09-10T19:04:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.2"}]: dispatch 2024-09-10T19:04:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.2"}]: dispatch 2024-09-10T19:04:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.3"}]: dispatch 2024-09-10T19:04:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.3"}]: dispatch 2024-09-10T19:04:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.4"}]: dispatch 2024-09-10T19:04:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.4"}]: dispatch 2024-09-10T19:04:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.5"}]: dispatch 2024-09-10T19:04:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.5"}]: dispatch 2024-09-10T19:04:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.6"}]: dispatch 2024-09-10T19:04:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.6"}]: dispatch 2024-09-10T19:04:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.7"}]: dispatch 2024-09-10T19:04:12.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.7"}]: dispatch 2024-09-10T19:04:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.8"}]: dispatch 2024-09-10T19:04:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.8"}]: dispatch 2024-09-10T19:04:12.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.9"}]: dispatch 2024-09-10T19:04:12.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi119-79536-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:12.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2777964774' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:12.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:12.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:12.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.0"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.0"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.1"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.1"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.2"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.2"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.3"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.3"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.4"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.4"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.5"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.5"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.6"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.6"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.7"}]: dispatch 2024-09-10T19:04:12.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.7"}]: dispatch 2024-09-10T19:04:12.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.8"}]: dispatch 2024-09-10T19:04:12.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.8"}]: dispatch 2024-09-10T19:04:12.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.9"}]: dispatch 2024-09-10T19:04:12.321 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.9"}]: dispatch 2024-09-10T19:04:12.321 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-09-10T19:04:12.321 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4244195218' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-09-10T19:04:12.321 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: twoxx 2024-09-10T19:04:12.321 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi119-79536-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:12.321 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:12.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:12.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: pgmap v69: 588 pgs: 232 unknown, 128 creating+peering, 228 active+clean; 584 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 16 KiB/s wr, 29 op/s 2024-09-10T19:04:12.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:12.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:12.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:12.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:12.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi119-79868-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:12.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:12.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:12.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:12.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1126701021' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T19:04:12.322 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:12.323 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1548473117' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:12.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.9"}]: dispatch 2024-09-10T19:04:12.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-09-10T19:04:12.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4244195218' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-09-10T19:04:12.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: twoxx 2024-09-10T19:04:12.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi119-79536-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:12.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:12.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:12.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: pgmap v69: 588 pgs: 232 unknown, 128 creating+peering, 228 active+clean; 584 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 16 KiB/s wr, 29 op/s 2024-09-10T19:04:12.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:12.324 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:12.324 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:12.324 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:12.324 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi119-79868-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:12.324 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:12.324 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:12.324 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:12.324 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1126701021' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-09-10T19:04:12.324 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:12.324 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1548473117' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:12.325 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.325 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1013785956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.325 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:12.325 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi119-79868-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:12.325 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:12.325 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: osdmap e71: 8 total, 8 up, 8 in 2024-09-10T19:04:12.325 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi119-79868-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:12.325 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi119-80015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:12.325 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-09-10T19:04:12.325 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:12.325 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.326 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1013785956' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:12.326 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:12.326 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi119-79868-6", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:12.326 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:12.326 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: osdmap e71: 8 total, 8 up, 8 in 2024-09-10T19:04:12.326 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi119-79868-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:12.326 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi119-80015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:12.326 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-09-10T19:04:12.327 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:13.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:04:12] "GET /metrics HTTP/1.1" 200 225518 "" "Prometheus/2.43.0" 2024-09-10T19:04:13.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: 21.6 deep-scrub starts 2024-09-10T19:04:13.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: 21.6 deep-scrub ok 2024-09-10T19:04:13.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: 21.8 deep-scrub starts 2024-09-10T19:04:13.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: 21.8 deep-scrub ok 2024-09-10T19:04:13.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi119-79573-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-10T19:04:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi119-79573-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi119-79512-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi119-79512-16"}]': finished 2024-09-10T19:04:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi119-79536-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi119-79536-22"}]': finished 2024-09-10T19:04:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:13.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:13.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:13.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:13.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi119-79573-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:13.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: osdmap e72: 8 total, 8 up, 8 in 2024-09-10T19:04:13.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi119-79610-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:13.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi119-79573-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:13.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:13.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1013785956' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi119-79499-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T19:04:13.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:13.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: 21.6 deep-scrub starts 2024-09-10T19:04:13.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: 21.6 deep-scrub ok 2024-09-10T19:04:13.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: 21.8 deep-scrub starts 2024-09-10T19:04:13.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: 21.8 deep-scrub ok 2024-09-10T19:04:13.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:13.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:13.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:13.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.070 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:13.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi119-79573-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-10T19:04:13.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi119-79573-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi119-79512-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi119-79512-16"}]': finished 2024-09-10T19:04:13.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi119-79536-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi119-79536-22"}]': finished 2024-09-10T19:04:13.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:13.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:13.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:13.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:13.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi119-79573-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:13.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: osdmap e72: 8 total, 8 up, 8 in 2024-09-10T19:04:13.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi119-79610-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:13.071 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi119-79573-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:13.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:13.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1013785956' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi119-79499-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T19:04:13.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:13.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: 21.6 deep-scrub starts 2024-09-10T19:04:13.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: 21.6 deep-scrub ok 2024-09-10T19:04:13.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: 21.8 deep-scrub starts 2024-09-10T19:04:13.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: 21.8 deep-scrub ok 2024-09-10T19:04:13.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:13.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:13.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:13.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:13.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi119-79573-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-10T19:04:13.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi119-79573-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi119-79512-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi119-79512-16"}]': finished 2024-09-10T19:04:13.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi119-79536-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi119-79536-22"}]': finished 2024-09-10T19:04:13.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:13.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:13.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:13.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:13.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi119-79573-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:13.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: osdmap e72: 8 total, 8 up, 8 in 2024-09-10T19:04:13.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi119-79610-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:13.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi119-79573-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:13.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:13.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1013785956' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi119-79499-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T19:04:13.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:13.421 INFO:tasks.workunit.client.0.smithi119.stdout: api_ser delete_pools_parallel: process_1_[80284]: starting. 2024-09-10T19:04:13.421 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_1_[80284]: creating pool ceph_test_rados_delete_pools_parallel.smithi119-80272 2024-09-10T19:04:13.421 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_1_[80284]: created object 0... 2024-09-10T19:04:13.421 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_1_[80284]: created object 25... 2024-09-10T19:04:13.421 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_1_[80284]: created object 49... 2024-09-10T19:04:13.421 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_1_[80284]: finishing. 2024-09-10T19:04:13.421 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_1_[80284]: shutting down. 2024-09-10T19:04:13.421 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_2_[80286]: starting. 2024-09-10T19:04:13.422 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_2_[80286]: deleting pool ceph_test_rados_delete_pools_parallel.smithi119-80272 2024-09-10T19:04:13.422 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_2_[80286]: shutting down. 2024-09-10T19:04:13.422 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: ******************************* 2024-09-10T19:04:13.422 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_3_[81341]: starting. 2024-09-10T19:04:13.422 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_3_[81341]: creating pool ceph_test_rados_delete_pools_parallel.smithi119-80272 2024-09-10T19:04:13.422 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_3_[81341]: created object 0... 2024-09-10T19:04:13.422 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_3_[81341]: created object 25... 2024-09-10T19:04:13.422 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_3_[81341]: created object 49... 2024-09-10T19:04:13.422 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_3_[81341]: finishing. 2024-09-10T19:04:13.422 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_3_[81341]: shutting down. 2024-09-10T19:04:13.422 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: ******************************* 2024-09-10T19:04:13.422 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_5_[81343]: starting. 2024-09-10T19:04:13.422 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_5_[81343]: listing objects. 2024-09-10T19:04:13.423 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_5_[81343]: listed object 0... 2024-09-10T19:04:13.423 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_5_[81343]: listed object 25... 2024-09-10T19:04:13.423 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_5_[81343]: saw 50 objects 2024-09-10T19:04:13.423 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_5_[81343]: shutting down. 2024-09-10T19:04:13.423 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: ******************************* 2024-09-10T19:04:13.423 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_4_[81342]: starting. 2024-09-10T19:04:13.423 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_4_[81342]: deleting pool ceph_test_rados_delete_pools_parallel.smithi119-80272 2024-09-10T19:04:13.423 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: process_4_[81342]: shutting down. 2024-09-10T19:04:13.423 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: ******************************* 2024-09-10T19:04:13.423 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: ******************************* 2024-09-10T19:04:13.423 INFO:tasks.workunit.client.0.smithi119.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-10T19:04:13.499 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: Running main() from gmock_main.cc 2024-09-10T19:04:13.499 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-09-10T19:04:13.499 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-10T19:04:13.500 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-09-10T19:04:13.500 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: seed 79536 2024-09-10T19:04:13.500 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-10T19:04:13.500 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-10T19:04:13.500 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-10T19:04:13.500 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (93 ms) 2024-09-10T19:04:13.500 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-10T19:04:13.500 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (13 ms) 2024-09-10T19:04:13.500 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-10T19:04:13.500 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (4 ms) 2024-09-10T19:04:13.500 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-10T19:04:13.500 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (4 ms) 2024-09-10T19:04:13.500 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-10T19:04:13.501 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (4 ms) 2024-09-10T19:04:13.501 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-10T19:04:13.501 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (5 ms) 2024-09-10T19:04:13.501 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-10T19:04:13.501 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (4 ms) 2024-09-10T19:04:13.501 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-10T19:04:13.501 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (6 ms) 2024-09-10T19:04:13.501 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-10T19:04:13.501 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (2 ms) 2024-09-10T19:04:13.501 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-10T19:04:13.501 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (5 ms) 2024-09-10T19:04:13.501 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-10T19:04:13.501 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (5 ms) 2024-09-10T19:04:13.502 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-10T19:04:13.502 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (4 ms) 2024-09-10T19:04:13.502 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-10T19:04:13.502 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (2 ms) 2024-09-10T19:04:13.502 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-10T19:04:13.502 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (2 ms) 2024-09-10T19:04:13.502 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-10T19:04:13.502 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (7 ms) 2024-09-10T19:04:13.502 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-10T19:04:13.502 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (3 ms) 2024-09-10T19:04:13.502 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-10T19:04:13.502 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (3 ms) 2024-09-10T19:04:13.502 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-10T19:04:13.502 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (2 ms) 2024-09-10T19:04:13.503 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-10T19:04:13.503 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (3 ms) 2024-09-10T19:04:13.503 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (171 ms total) 2024-09-10T19:04:13.503 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: 2024-09-10T19:04:13.503 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-09-10T19:04:13.503 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-10T19:04:13.503 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1072 ms) 2024-09-10T19:04:13.503 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-10T19:04:13.503 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (11 ms) 2024-09-10T19:04:13.503 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-10T19:04:13.503 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (5 ms) 2024-09-10T19:04:13.503 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-10T19:04:13.503 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (5 ms) 2024-09-10T19:04:13.504 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-10T19:04:13.504 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (5 ms) 2024-09-10T19:04:13.912 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] L "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:13.912 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:10.374354+0000 mon.a [INF] from='client.? 172.21.15.119:0/1548473117' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:13.912 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:11.901692+0000 mon.c [INF] from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.912 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:11.901920+0000 mon.a [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:13.912 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:11.902861+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:13.913 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:11.903164+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:13.913 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:11.906506+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.913 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:11.906788+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:13.913 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:11.907431+0000 mon.b [INF] from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi119-79573-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.913 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:11.910712+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-10T19:04:13.913 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:11.913261+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi119-79573-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:13.913 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.397293+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi119-79512-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi119-79512-16"}]': finished 2024-09-10T19:04:13.913 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.397380+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi119-79536-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi119-79536-22"}]': finished 2024-09-10T19:04:13.913 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.903407+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:13.913 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.903482+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi119-79499-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:14.156 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.903589+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10", "profile" open_pools_parallel: process_1_[80321]: starting. 2024-09-10T19:04:14.156 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_1_[80321]: creating pool ceph_test_rados_open_pools_parallel.smithi119-80298 2024-09-10T19:04:14.156 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_1_[80321]: created object 0... 2024-09-10T19:04:14.157 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_1_[80321]: created object 25... 2024-09-10T19:04:14.157 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_1_[80321]: created object 49... 2024-09-10T19:04:14.157 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_1_[80321]: finishing. 2024-09-10T19:04:14.157 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_1_[80321]: shutting down. 2024-09-10T19:04:14.157 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_2_[80322]: starting. 2024-09-10T19:04:14.157 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_2_[80322]: rados_pool_create. 2024-09-10T19:04:14.157 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_2_[80322]: rados_ioctx_create. 2024-09-10T19:04:14.157 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_2_[80322]: shutting down. 2024-09-10T19:04:14.157 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: ******************************* 2024-09-10T19:04:14.157 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_3_[81338]: starting. 2024-09-10T19:04:14.157 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_3_[81338]: creating pool ceph_test_rados_open_pools_parallel.smithi119-80298 2024-09-10T19:04:14.157 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_3_[81338]: created object 0... 2024-09-10T19:04:14.158 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_3_[81338]: created object 25... 2024-09-10T19:04:14.158 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_3_[81338]: created object 49... 2024-09-10T19:04:14.158 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_3_[81338]: finishing. 2024-09-10T19:04:14.158 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_3_[81338]: shutting down. 2024-09-10T19:04:14.158 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: ******************************* 2024-09-10T19:04:14.158 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_4_[81339]: starting. 2024-09-10T19:04:14.158 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_4_[81339]: rados_pool_create. 2024-09-10T19:04:14.158 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_4_[81339]: rados_ioctx_create. 2024-09-10T19:04:14.158 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: process_4_[81339]: shutting down. 2024-09-10T19:04:14.158 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: ******************************* 2024-09-10T19:04:14.158 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: ******************************* 2024-09-10T19:04:14.158 INFO:tasks.workunit.client.0.smithi119.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-10T19:04:14.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:14 smithi194 ceph-mon[32464]: 21.0 deep-scrub starts 2024-09-10T19:04:14.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:14 smithi194 ceph-mon[32464]: 21.0 deep-scrub ok 2024-09-10T19:04:14.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi119-79610-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:14.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi119-79499-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T19:04:14.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:14.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi119-79573-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:14.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:14.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:14 smithi119 ceph-mon[40674]: 21.0 deep-scrub starts 2024-09-10T19:04:14.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:14 smithi119 ceph-mon[40674]: 21.0 deep-scrub ok 2024-09-10T19:04:14.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi119-79610-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:14.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi119-79499-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T19:04:14.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:14.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi119-79573-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:14.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:14.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:14 smithi119 ceph-mon[29577]: 21.0 deep-scrub starts 2024-09-10T19:04:14.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:14 smithi119 ceph-mon[29577]: 21.0 deep-scrub ok 2024-09-10T19:04:14.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi119-79610-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:14.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi119-79499-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T19:04:14.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:14.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi119-79573-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:14.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:15.156 INFO:tasks.workunit.client.0.smithi119.stdout:: [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:15.157 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.903842+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi119-79573-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:15.157 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.904623+0000 mon.c [INF] from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi119-79610-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:15.157 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.905247+0000 mon.b [INF] from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi119-79573-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:15.157 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.905447+0000 mon.b [INF] from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:15.157 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.905465+0000 mon.c [INF] from='client.? 172.21.15.119:0/1013785956' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi119-79499-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T19:04:15.157 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.905750+0000 mon.c [INF] from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:15.157 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.918554+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi119-79610-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:15.157 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.919172+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi119-79499-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-09-10T19:04:15.157 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.919386+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:15.157 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.921304+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi119-79573-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:15.157 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:12.921566+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:15.157 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:13.411729+0000 mon.a [INF] from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi119-79868-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi119-79868-6"}]': finished 2024-09-10T19:04:15.158 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:13.411814+0000 mon.a [INF] from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi119-80015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi119-80015-7"}]': finished 2024-09-10T19:04:15.158 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:13.675231+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi119-79499-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-10T19:04:15.158 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:13.675316+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]': finished 2024-09-10T19:04:15.158 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:13.675897+0000 mon.c [INF] from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-4", "overlaypool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:15.158 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:13.911614+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]': finished 2024-09-10T19:04:15.158 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:13.911756+0000 mon.c [INF] from='client.? 172.21.15.119:0/4244195218' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-10T19:04:15.158 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:13.912335+0000 mon.b [INF] from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:15.158 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:13.912451+0000 mon.c [INF] from='client.? 172.21.15.119:0/3184146204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.158 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:13.912619+0000 mon.c [INF] from='client.? 172.21.15.119:0/943481859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi119-79477-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.158 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:13.912635+0000 client.admin [INF] threexx 2024-09-10T19:04:15.158 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.156589+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-4", "overlaypool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:15.158 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.393383+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-10T19:04:15.158 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.393602+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.159 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.393803+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi119-79477-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.159 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.393945+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:15.159 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.414224+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi119-79610-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi119-79610-10"}]': finished 2024-09-10T19:04:15.159 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.414331+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi119-79573-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi119-79573-10"}]': finished 2024-09-10T19:04:15.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: 21.2 deep-scrub starts 2024-09-10T19:04:15.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: 21.2 deep-scrub ok 2024-09-10T19:04:15.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: 21.3 deep-scrub starts 2024-09-10T19:04:15.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: 21.3 deep-scrub ok 2024-09-10T19:04:15.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: pgmap v72: 596 pgs: 16 creating+peering, 580 active+clean; 145 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 542 op/s 2024-09-10T19:04:15.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: 21.9 deep-scrub starts 2024-09-10T19:04:15.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi119-79868-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi119-79868-6"}]': finished 2024-09-10T19:04:15.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi119-80015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi119-80015-7"}]': finished 2024-09-10T19:04:15.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: 21.9 deep-scrub ok 2024-09-10T19:04:15.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: 21.7 deep-scrub starts 2024-09-10T19:04:15.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: 21.7 deep-scrub ok 2024-09-10T19:04:15.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:15.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi119-79499-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-10T19:04:15.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]': finished 2024-09-10T19:04:15.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-4", "overlaypool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:15.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]': finished 2024-09-10T19:04:15.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4244195218' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-10T19:04:15.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:15.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3184146204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/943481859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi119-79477-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: threexx 2024-09-10T19:04:15.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: 21.1 deep-scrub starts 2024-09-10T19:04:15.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: 21.1 deep-scrub ok 2024-09-10T19:04:15.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: osdmap e73: 8 total, 8 up, 8 in 2024-09-10T19:04:15.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-4", "overlaypool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:15.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-10T19:04:15.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi119-79477-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:15.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi119-79610-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi119-79610-10"}]': finished 2024-09-10T19:04:15.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi119-79573-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi119-79573-10"}]': finished 2024-09-10T19:04:15.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-4", "overlaypool": "test-rados-api-smithi119-79782-4-cache"}]': finished 2024-09-10T19:04:15.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:15.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi119-79477-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:15.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]': finished 2024-09-10T19:04:15.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: osdmap e74: 8 total, 8 up, 8 in 2024-09-10T19:04:15.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2746093865' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:15.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:15.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-4-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:15.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:15.523 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.523 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:15.523 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.523 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-4-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:15.523 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: 21.2 deep-scrub starts 2024-09-10T19:04:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: 21.2 deep-scrub ok 2024-09-10T19:04:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: 21.3 deep-scrub starts 2024-09-10T19:04:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: 21.3 deep-scrub ok 2024-09-10T19:04:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: pgmap v72: 596 pgs: 16 creating+peering, 580 active+clean; 145 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 542 op/s 2024-09-10T19:04:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: 21.9 deep-scrub starts 2024-09-10T19:04:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi119-79868-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi119-79868-6"}]': finished 2024-09-10T19:04:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi119-80015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi119-80015-7"}]': finished 2024-09-10T19:04:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: 21.9 deep-scrub ok 2024-09-10T19:04:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: 21.7 deep-scrub starts 2024-09-10T19:04:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: 21.7 deep-scrub ok 2024-09-10T19:04:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi119-79499-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-10T19:04:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]': finished 2024-09-10T19:04:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-4", "overlaypool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]': finished 2024-09-10T19:04:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4244195218' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-10T19:04:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3184146204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/943481859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi119-79477-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: threexx 2024-09-10T19:04:15.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: 21.1 deep-scrub starts 2024-09-10T19:04:15.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: 21.1 deep-scrub ok 2024-09-10T19:04:15.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: osdmap e73: 8 total, 8 up, 8 in 2024-09-10T19:04:15.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-4", "overlaypool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:15.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-10T19:04:15.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: 21.2 deep-scrub starts 2024-09-10T19:04:15.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: 21.2 deep-scrub ok 2024-09-10T19:04:15.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: 21.3 deep-scrub starts 2024-09-10T19:04:15.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: 21.3 deep-scrub ok 2024-09-10T19:04:15.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: pgmap v72: 596 pgs: 16 creating+peering, 580 active+clean; 145 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 542 op/s 2024-09-10T19:04:15.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: 21.9 deep-scrub starts 2024-09-10T19:04:15.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi119-79868-6", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi119-79868-6"}]': finished 2024-09-10T19:04:15.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi119-80015-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi119-80015-7"}]': finished 2024-09-10T19:04:15.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: 21.9 deep-scrub ok 2024-09-10T19:04:15.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: 21.7 deep-scrub starts 2024-09-10T19:04:15.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: 21.7 deep-scrub ok 2024-09-10T19:04:15.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:15.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi119-79499-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-10T19:04:15.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]': finished 2024-09-10T19:04:15.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-4", "overlaypool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:15.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi119-80068-1"}]': finished 2024-09-10T19:04:15.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4244195218' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-10T19:04:15.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1534584943' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:15.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3184146204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/943481859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi119-79477-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: threexx 2024-09-10T19:04:15.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: 21.1 deep-scrub starts 2024-09-10T19:04:15.571 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: 21.1 deep-scrub ok 2024-09-10T19:04:15.572 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: osdmap e73: 8 total, 8 up, 8 in 2024-09-10T19:04:15.572 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-4", "overlaypool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:15.572 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-09-10T19:04:15.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi119-79477-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:15.572 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi119-79610-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi119-79610-10"}]': finished 2024-09-10T19:04:15.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi119-79573-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi119-79573-10"}]': finished 2024-09-10T19:04:15.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-4", "overlaypool": "test-rados-api-smithi119-79782-4-cache"}]': finished 2024-09-10T19:04:15.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:15.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi119-79477-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:15.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]': finished 2024-09-10T19:04:15.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: osdmap e74: 8 total, 8 up, 8 in 2024-09-10T19:04:15.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2746093865' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:15.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:15.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-4-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:15.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:15.573 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.574 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:15.574 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.574 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-4-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:15.574 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:15.574 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.574 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi119-79477-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.574 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]: dispatch 2024-09-10T19:04:15.574 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi119-79610-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi119-79610-10"}]': finished 2024-09-10T19:04:15.574 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi119-79573-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi119-79573-10"}]': finished 2024-09-10T19:04:15.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-4", "overlaypool": "test-rados-api-smithi119-79782-4-cache"}]': finished 2024-09-10T19:04:15.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:15.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi119-79477-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:15.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]': finished 2024-09-10T19:04:15.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: osdmap e74: 8 total, 8 up, 8 in 2024-09-10T19:04:15.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2746093865' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:15.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:15.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-4-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:15.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:15.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.575 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:15.576 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:15.576 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-4-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:15.576 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:16.429 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.414602+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-4", "overlaypool": "test-rados-api-smithi119-79782-4-cache"}]': fi api_io: Running main() from gmock_main.cc 2024-09-10T19:04:16.429 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-10T19:04:16.429 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [----------] Global test environment set-up. 2024-09-10T19:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-10T19:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-10T19:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (113 ms) 2024-09-10T19:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-10T19:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-10T19:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-10T19:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: no timeout :/ 2024-09-10T19:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: no timeout :/ 2024-09-10T19:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: no timeout :/ 2024-09-10T19:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: no timeout :/ 2024-09-10T19:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: no timeout :/ 2024-09-10T19:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (198 ms) 2024-09-10T19:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-10T19:04:16.430 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (10 ms) 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIo.Checksum (4 ms) 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (4 ms) 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (2 ms) 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (6 ms) 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (1 ms) 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIo.TruncTest (2 ms) 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (2 ms) 2024-09-10T19:04:16.431 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-10T19:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (2 ms) 2024-09-10T19:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-10T19:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIo.RmXattr (6 ms) 2024-09-10T19:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-10T19:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIo.XattrIter (3 ms) 2024-09-10T19:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [----------] 14 tests from LibRadosIo (353 ms total) 2024-09-10T19:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: 2024-09-10T19:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-10T19:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-10T19:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1073 ms) 2024-09-10T19:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-10T19:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (5 ms) 2024-09-10T19:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-10T19:04:16.432 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (4 ms) 2024-09-10T19:04:16.433 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-10T19:04:16.433 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (4 ms) 2024-09-10T19:04:16.433 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-10T19:04:16.433 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (8 ms) 2024-09-10T19:04:16.433 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-10T19:04:16.433 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (4 ms) 2024-09-10T19:04:16.433 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-10T19:04:16.433 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (2 ms) 2024-09-10T19:04:16.433 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-10T19:04:16.433 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (3 ms) 2024-09-10T19:04:16.433 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-10T19:04:16.433 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (9 ms) 2024-09-10T19:04:16.433 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-10T19:04:16.434 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-09-10T19:04:16.434 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1116 ms total) 2024-09-10T19:04:16.434 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: 2024-09-10T19:04:16.434 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [----------] Global test environment tear-down 2024-09-10T19:04:16.434 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [==========] 24 tests from 2 test suites ran. (15171 ms total) 2024-09-10T19:04:16.434 INFO:tasks.workunit.client.0.smithi119.stdout: api_io: [ PASSED ] 24 tests. 2024-09-10T19:04:16.435 INFO:tasks.workunit.client.0.smithi119.stdout:ibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-10T19:04:16.435 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (3 ms) 2024-09-10T19:04:16.435 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-10T19:04:16.435 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (6 ms) 2024-09-10T19:04:16.435 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-10T19:04:16.436 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (4 ms) 2024-09-10T19:04:16.436 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-10T19:04:16.436 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (6 ms) 2024-09-10T19:04:16.436 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-10T19:04:16.436 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (4 ms) 2024-09-10T19:04:16.436 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-10T19:04:16.436 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (2 ms) 2024-09-10T19:04:16.436 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-10T19:04:16.436 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-09-10T19:04:16.436 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-10T19:04:16.436 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (6 ms) 2024-09-10T19:04:16.436 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-10T19:04:16.436 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (4 ms) 2024-09-10T19:04:16.437 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-10T19:04:16.437 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (5 ms) 2024-09-10T19:04:16.437 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-10T19:04:16.437 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (5 ms) 2024-09-10T19:04:16.437 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-10T19:04:16.437 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (6 ms) 2024-09-10T19:04:16.437 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (1152 ms total) 2024-09-10T19:04:16.437 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: 2024-09-10T19:04:16.437 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-10T19:04:16.437 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (15170 ms total) 2024-09-10T19:04:16.437 INFO:tasks.workunit.client.0.smithi119.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-09-10T19:04:16.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: pgmap v75: 644 pgs: 160 unknown, 484 active+clean; 145 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 520 op/s 2024-09-10T19:04:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-09-10T19:04:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4244195218' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-09-10T19:04:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: fourxx 2024-09-10T19:04:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-09-10T19:04:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]': finished 2024-09-10T19:04:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]': finished 2024-09-10T19:04:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-4-cache", "mode": "writeback"}]': finished 2024-09-10T19:04:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: osdmap e75: 8 total, 8 up, 8 in 2024-09-10T19:04:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:16.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:16.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:16.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:16.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:16.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:16.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:16.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-4"}]: dispatch 2024-09-10T19:04:16.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-09-10T19:04:16.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:16.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: 21.4 deep-scrub starts 2024-09-10T19:04:16.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: 21.4 deep-scrub ok 2024-09-10T19:04:16.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:16.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-4"}]: dispatch 2024-09-10T19:04:16.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: pgmap v75: 644 pgs: 160 unknown, 484 active+clean; 145 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 520 op/s 2024-09-10T19:04:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-09-10T19:04:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4244195218' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-09-10T19:04:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: fourxx 2024-09-10T19:04:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-09-10T19:04:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]': finished 2024-09-10T19:04:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]': finished 2024-09-10T19:04:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-4-cache", "mode": "writeback"}]': finished 2024-09-10T19:04:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: osdmap e75: 8 total, 8 up, 8 in 2024-09-10T19:04:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-4"}]: dispatch 2024-09-10T19:04:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-09-10T19:04:16.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:16.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: 21.4 deep-scrub starts 2024-09-10T19:04:16.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: 21.4 deep-scrub ok 2024-09-10T19:04:16.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:16.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-4"}]: dispatch 2024-09-10T19:04:16.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:16.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: pgmap v75: 644 pgs: 160 unknown, 484 active+clean; 145 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 520 op/s 2024-09-10T19:04:16.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-09-10T19:04:16.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4244195218' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-09-10T19:04:16.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: fourxx 2024-09-10T19:04:16.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-09-10T19:04:16.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:16.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]': finished 2024-09-10T19:04:16.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:16.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]': finished 2024-09-10T19:04:16.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:16.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-4-cache", "mode": "writeback"}]': finished 2024-09-10T19:04:16.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: osdmap e75: 8 total, 8 up, 8 in 2024-09-10T19:04:16.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:16.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:16.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:16.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:16.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:16.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:16.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:16.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:16.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-4"}]: dispatch 2024-09-10T19:04:16.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-09-10T19:04:16.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:16.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: 21.4 deep-scrub starts 2024-09-10T19:04:16.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: 21.4 deep-scrub ok 2024-09-10T19:04:16.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:16.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-4"}]: dispatch 2024-09-10T19:04:16.571 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.116 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-10T19:04:17.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-10T19:04:17.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-10T19:04:17.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-10T19:04:17.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: seed 80015 2024-09-10T19:04:17.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-10T19:04:17.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (235 ms) 2024-09-10T19:04:17.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-10T19:04:17.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (4 ms) 2024-09-10T19:04:17.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-10T19:04:17.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (1 ms) 2024-09-10T19:04:17.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-10T19:04:17.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (2 ms) 2024-09-10T19:04:17.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-10T19:04:17.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (11 ms) 2024-09-10T19:04:17.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (253 ms total) 2024-09-10T19:04:17.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: 2024-09-10T19:04:17.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-10T19:04:17.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-10T19:04:17.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (325 ms) 2024-09-10T19:04:17.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-10T19:04:17.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (239 ms) 2024-09-10T19:04:17.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-10T19:04:17.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (2 ms) 2024-09-10T19:04:17.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-10T19:04:17.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (6 ms) 2024-09-10T19:04:17.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (572 ms total) 2024-09-10T19:04:17.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: 2024-09-10T19:04:17.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-10T19:04:17.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (15703 ms total) 2024-09-10T19:04:17.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-10T19:04:17.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: Running main() from gmock_main.cc 2024-09-10T19:04:17.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [==========] Running 8 tests from 2 test suites. 2024-09-10T19:04:17.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [----------] Global test environment set-up. 2024-09-10T19:04:17.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [----------] 4 tests from LibRadosStat 2024-09-10T19:04:17.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-10T19:04:17.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ OK ] LibRadosStat.Stat (224 ms) 2024-09-10T19:04:17.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-10T19:04:17.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ OK ] LibRadosStat.StatNS (16 ms) 2024-09-10T19:04:17.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-10T19:04:17.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-09-10T19:04:17.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-10T19:04:17.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (5 ms) 2024-09-10T19:04:17.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [----------] 4 tests from LibRadosStat (245 ms total) 2024-09-10T19:04:17.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: 2024-09-10T19:04:17.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-10T19:04:17.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-10T19:04:17.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (327 ms) 2024-09-10T19:04:17.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-10T19:04:17.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (6 ms) 2024-09-10T19:04:17.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-10T19:04:17.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (231 ms) 2024-09-10T19:04:17.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-10T19:04:17.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (2 ms) 2024-09-10T19:04:17.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (566 ms total) 2024-09-10T19:04:17.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: 2024-09-10T19:04:17.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [----------] Global test environment tear-down 2024-09-10T19:04:17.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [==========] 8 tests from 2 test suites ran. (15744 ms total) 2024-09-10T19:04:17.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_stat: [ PASSED ] 8 tests. 2024-09-10T19:04:17.471 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-09-10T19:04:17.471 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]': finished 2024-09-10T19:04:17.471 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]': finished 2024-09-10T19:04:17.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7"}]': finished 2024-09-10T19:04:17.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi119-79868-6"}]': finished 2024-09-10T19:04:17.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-4"}]': finished 2024-09-10T19:04:17.472 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.473 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: osdmap e76: 8 total, 8 up, 8 in 2024-09-10T19:04:17.473 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:17.473 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:17.473 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app2"}]: dispatch 2024-09-10T19:04:17.473 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.473 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2596733817' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.474 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:17.474 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:17.474 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.474 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/677498115' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi119-79477-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.474 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.474 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pg_num": 4}]: dispatch 2024-09-10T19:04:17.474 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pg_num": 4}]: dispatch 2024-09-10T19:04:17.475 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:17.475 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:04:17.475 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:17.475 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:17.475 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:17.475 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi119-80015-7"}]': finished 2024-09-10T19:04:17.476 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]': finished 2024-09-10T19:04:17.476 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi119-79868-6"}]': finished 2024-09-10T19:04:17.476 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.476 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/677498115' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi119-79477-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.476 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.476 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pg_num": 4}]': finished 2024-09-10T19:04:17.477 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: osdmap e77: 8 total, 8 up, 8 in 2024-09-10T19:04:17.477 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:17.477 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-09-10T19:04:17.477 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2153469343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.477 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:04:17.477 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:17.478 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-09-10T19:04:17.478 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]': finished 2024-09-10T19:04:17.478 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]': finished 2024-09-10T19:04:17.479 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.479 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7"}]': finished 2024-09-10T19:04:17.479 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi119-79868-6"}]': finished 2024-09-10T19:04:17.479 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-4"}]': finished 2024-09-10T19:04:17.479 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.479 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: osdmap e76: 8 total, 8 up, 8 in 2024-09-10T19:04:17.480 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:17.480 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:17.480 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app2"}]: dispatch 2024-09-10T19:04:17.480 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.480 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2596733817' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.481 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:17.481 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:17.481 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.481 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/677498115' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi119-79477-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.481 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.482 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:04:17.483 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pg_num": 4}]: dispatch 2024-09-10T19:04:17.483 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pg_num": 4}]: dispatch 2024-09-10T19:04:17.483 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:17.483 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:04:17.483 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:17.483 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:17.484 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:17.484 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi119-80015-7"}]': finished 2024-09-10T19:04:17.484 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]': finished 2024-09-10T19:04:17.484 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi119-79868-6"}]': finished 2024-09-10T19:04:17.484 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.484 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/677498115' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi119-79477-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.485 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.485 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pg_num": 4}]': finished 2024-09-10T19:04:17.485 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: osdmap e77: 8 total, 8 up, 8 in 2024-09-10T19:04:17.485 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:17.485 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-09-10T19:04:17.485 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2153469343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.486 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:04:17.486 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:17.486 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.486 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:04:17.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-09-10T19:04:17.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi119-79536-22"}]': finished 2024-09-10T19:04:17.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi119-79512-16"}]': finished 2024-09-10T19:04:17.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi119-80015-7"}]': finished 2024-09-10T19:04:17.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi119-79868-6"}]': finished 2024-09-10T19:04:17.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-4"}]': finished 2024-09-10T19:04:17.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: osdmap e76: 8 total, 8 up, 8 in 2024-09-10T19:04:17.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi119-80015-7"}]: dispatch 2024-09-10T19:04:17.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:17.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app2"}]: dispatch 2024-09-10T19:04:17.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2596733817' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]: dispatch 2024-09-10T19:04:17.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi119-79868-6"}]: dispatch 2024-09-10T19:04:17.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/677498115' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi119-79477-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pg_num": 4}]: dispatch 2024-09-10T19:04:17.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pg_num": 4}]: dispatch 2024-09-10T19:04:17.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:17.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:04:17.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:17.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:17.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:17.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/945406467' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi119-80015-7"}]': finished 2024-09-10T19:04:17.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-4", "tierpool": "test-rados-api-smithi119-79782-4-cache"}]': finished 2024-09-10T19:04:17.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3598166492' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi119-79868-6"}]': finished 2024-09-10T19:04:17.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/677498115' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi119-79477-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:17.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pg_num": 4}]': finished 2024-09-10T19:04:17.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: osdmap e77: 8 total, 8 up, 8 in 2024-09-10T19:04:17.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:17.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-09-10T19:04:17.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2153469343' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:04:17.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:17.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:17.522 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:04:18.175 INFO:tasks.workunit.client.0.smithi119.stdout:nished 2024-09-10T19:04:18.175 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.414748+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:18.175 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.414826+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi119-79477-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:18.176 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.414900+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi119-80068-1"}]': finished 2024-09-10T19:04:18.176 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.417556+0000 mon.c [INF] from='client.? 172.21.15.119:0/2746093865' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:18.176 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.417725+0000 mon.b [INF] from='client.? 172.21.15.119:0/2737245746' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:18.176 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.419731+0000 mon.c [INF] from='client.? 172.21.15.119:0/3094694895' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:18.176 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.420285+0000 mon.c [INF] from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:18.176 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.420589+0000 mon.c [INF] from='client.? 172.21.15.119:0/2898593651' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-4-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:18.176 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.422298+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi119-79536-22"}]: dispatch 2024-09-10T19:04:18.176 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.422483+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:18.176 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.422735+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi119-79512-16"}]: dispatch 2024-09-10T19:04:18.176 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.422986+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi119-79673-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:18.176 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: got: 2024-09-10T19:04:14.423174+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-4-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:04:18.177 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (8523 ms) 2024-09-10T19:04:18.177 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (16753 ms total) 2024-09-10T19:04:18.177 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: 2024-09-10T19:04:18.177 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-10T19:04:18.177 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (16753 ms total) 2024-09-10T19:04:18.177 INFO:tasks.workunit.client.0.smithi119.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-10T19:04:18.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:04:19.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-mon[32464]: pgmap v78: 628 pgs: 144 unknown, 484 active+clean; 145 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:19.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-10T19:04:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-mon[32464]: osdmap e78: 8 total, 8 up, 8 in 2024-09-10T19:04:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T19:04:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]: dispatch 2024-09-10T19:04:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T19:04:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]: dispatch 2024-09-10T19:04:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[29577]: pgmap v78: 628 pgs: 144 unknown, 484 active+clean; 145 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-10T19:04:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[29577]: osdmap e78: 8 total, 8 up, 8 in 2024-09-10T19:04:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T19:04:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]: dispatch 2024-09-10T19:04:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T19:04:19.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]: dispatch 2024-09-10T19:04:19.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:19.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:19.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[40674]: pgmap v78: 628 pgs: 144 unknown, 484 active+clean; 145 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-10T19:04:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[40674]: osdmap e78: 8 total, 8 up, 8 in 2024-09-10T19:04:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T19:04:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]: dispatch 2024-09-10T19:04:19.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:19.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T19:04:19.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]: dispatch 2024-09-10T19:04:19.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:19.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:19.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:20.115 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: Running main() from gmock_main.cc 2024-09-10T19:04:20.115 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-10T19:04:20.115 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [----------] Global test environment set-up. 2024-09-10T19:04:20.115 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-10T19:04:20.115 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-10T19:04:20.115 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (86 ms) 2024-09-10T19:04:20.115 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-10T19:04:20.115 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLock.LockShared (12 ms) 2024-09-10T19:04:20.115 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-10T19:04:20.115 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1117 ms) 2024-09-10T19:04:20.115 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-10T19:04:20.116 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1059 ms) 2024-09-10T19:04:20.116 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-10T19:04:20.116 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (17 ms) 2024-09-10T19:04:20.116 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-10T19:04:20.116 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLock.Unlock (4 ms) 2024-09-10T19:04:20.116 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-10T19:04:20.116 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (6 ms) 2024-09-10T19:04:20.116 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-10T19:04:20.116 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (4 ms) 2024-09-10T19:04:20.116 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [----------] 8 tests from LibRadosLock (2305 ms total) 2024-09-10T19:04:20.116 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: 2024-09-10T19:04:20.116 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-10T19:04:20.116 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-10T19:04:20.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1087 ms) 2024-09-10T19:04:20.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-10T19:04:20.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (5 ms) 2024-09-10T19:04:20.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-10T19:04:20.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1004 ms) 2024-09-10T19:04:20.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-10T19:04:20.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1004 ms) 2024-09-10T19:04:20.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-10T19:04:20.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (4 ms) 2024-09-10T19:04:20.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-10T19:04:20.117 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (4 ms) 2024-09-10T19:04:20.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-10T19:04:20.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (5 ms) 2024-09-10T19:04:20.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-10T19:04:20.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (3 ms) 2024-09-10T19:04:20.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3116 ms total) 2024-09-10T19:04:20.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: 2024-09-10T19:04:20.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [----------] Global test environment tear-down 2024-09-10T19:04:20.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (18833 ms total) 2024-09-10T19:04:20.118 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-10T19:04:20.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-10T19:04:20.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-10T19:04:20.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-10T19:04:20.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-10T19:04:20.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: seed 79610 2024-09-10T19:04:20.119 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-10T19:04:20.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (96 ms) 2024-09-10T19:04:20.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-10T19:04:20.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (6 ms) 2024-09-10T19:04:20.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-10T19:04:20.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1144 ms) 2024-09-10T19:04:20.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-10T19:04:20.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1042 ms) 2024-09-10T19:04:20.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-10T19:04:20.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (13 ms) 2024-09-10T19:04:20.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-10T19:04:20.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (4 ms) 2024-09-10T19:04:20.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-10T19:04:20.120 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (5 ms) 2024-09-10T19:04:20.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-10T19:04:20.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (3 ms) 2024-09-10T19:04:20.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2313 ms total) 2024-09-10T19:04:20.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: 2024-09-10T19:04:20.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-10T19:04:20.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-10T19:04:20.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1068 ms) 2024-09-10T19:04:20.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-10T19:04:20.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (18 ms) 2024-09-10T19:04:20.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-10T19:04:20.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1005 ms) 2024-09-10T19:04:20.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-10T19:04:20.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1005 ms) 2024-09-10T19:04:20.121 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-10T19:04:20.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (4 ms) 2024-09-10T19:04:20.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-10T19:04:20.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (3 ms) 2024-09-10T19:04:20.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-10T19:04:20.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (5 ms) 2024-09-10T19:04:20.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-10T19:04:20.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (4 ms) 2024-09-10T19:04:20.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3112 ms total) 2024-09-10T19:04:20.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: 2024-09-10T19:04:20.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-10T19:04:20.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (18817 ms total) 2024-09-10T19:04:20.122 INFO:tasks.workunit.client.0.smithi119.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-10T19:04:20.268 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:04:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:04:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: pgmap v81: 520 pgs: 36 creating+peering, 484 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 7.0 KiB/s wr, 5 op/s 2024-09-10T19:04:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-10T19:04:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]': finished 2024-09-10T19:04:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]': finished 2024-09-10T19:04:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]': finished 2024-09-10T19:04:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: osdmap e79: 8 total, 8 up, 8 in 2024-09-10T19:04:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "mode": "writeback"}]: dispatch 2024-09-10T19:04:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T19:04:20.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3352858953' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi119-79477-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:20.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4073602015' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi119-79823-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/851624313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:20.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "mode": "writeback"}]: dispatch 2024-09-10T19:04:20.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T19:04:20.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi119-79477-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:20.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi119-79823-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:20.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:20.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:20.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:20.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:20.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:20.271 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:20.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:20.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: pgmap v81: 520 pgs: 36 creating+peering, 484 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 7.0 KiB/s wr, 5 op/s 2024-09-10T19:04:20.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-10T19:04:20.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]': finished 2024-09-10T19:04:20.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]': finished 2024-09-10T19:04:20.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]': finished 2024-09-10T19:04:20.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: osdmap e79: 8 total, 8 up, 8 in 2024-09-10T19:04:20.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "mode": "writeback"}]: dispatch 2024-09-10T19:04:20.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:20.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T19:04:20.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3352858953' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi119-79477-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:20.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4073602015' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi119-79823-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/851624313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:20.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "mode": "writeback"}]: dispatch 2024-09-10T19:04:20.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T19:04:20.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi119-79477-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:20.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi119-79823-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:20.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:20.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:20.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:20.321 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:20.321 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:20.321 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:20.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:20.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:20.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: pgmap v81: 520 pgs: 36 creating+peering, 484 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 7.0 KiB/s wr, 5 op/s 2024-09-10T19:04:20.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-10T19:04:20.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "overlaypool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]': finished 2024-09-10T19:04:20.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi119-79573-10"}]': finished 2024-09-10T19:04:20.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi119-79610-10"}]': finished 2024-09-10T19:04:20.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: osdmap e79: 8 total, 8 up, 8 in 2024-09-10T19:04:20.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "mode": "writeback"}]: dispatch 2024-09-10T19:04:20.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4123372196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:20.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4152630087' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T19:04:20.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3352858953' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi119-79477-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/865421205' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:20.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4073602015' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi119-79823-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/851624313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]: dispatch 2024-09-10T19:04:20.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "mode": "writeback"}]: dispatch 2024-09-10T19:04:20.322 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T19:04:20.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi119-79477-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]: dispatch 2024-09-10T19:04:20.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi119-79823-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:20.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:20.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:20.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:20.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:20.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:20.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:20.323 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:21.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/851624313' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]': finished 2024-09-10T19:04:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "mode": "writeback"}]': finished 2024-09-10T19:04:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1"}]': finished 2024-09-10T19:04:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi119-79477-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]': finished 2024-09-10T19:04:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi119-79823-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: osdmap e80: 8 total, 8 up, 8 in 2024-09-10T19:04:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi119-79960-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3670857195' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi119-79960-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:21.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:21.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:21.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/851624313' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:21.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]': finished 2024-09-10T19:04:21.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "mode": "writeback"}]': finished 2024-09-10T19:04:21.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1"}]': finished 2024-09-10T19:04:21.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi119-79477-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:21.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]': finished 2024-09-10T19:04:21.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi119-79823-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:21.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:21.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: osdmap e80: 8 total, 8 up, 8 in 2024-09-10T19:04:21.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi119-79960-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:21.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3670857195' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:21.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi119-79960-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:21.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:21.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:21.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:21.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/851624313' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:21.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi119-79573-10"}]': finished 2024-09-10T19:04:21.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "mode": "writeback"}]': finished 2024-09-10T19:04:21.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi119-79673-1","app":"app1","key":"key1"}]': finished 2024-09-10T19:04:21.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi119-79477-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:21.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi119-79610-10"}]': finished 2024-09-10T19:04:21.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi119-79823-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:21.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:21.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: osdmap e80: 8 total, 8 up, 8 in 2024-09-10T19:04:21.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi119-79960-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:21.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3670857195' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:21.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi119-79960-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:21.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:21.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:22.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:22.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:22 smithi194 ceph-mon[32464]: osdmap e81: 8 total, 8 up, 8 in 2024-09-10T19:04:22.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1185087263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi119-79837-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi119-79837-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:22.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-mon[29577]: osdmap e81: 8 total, 8 up, 8 in 2024-09-10T19:04:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1185087263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi119-79837-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi119-79837-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:22.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:22.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-mon[40674]: osdmap e81: 8 total, 8 up, 8 in 2024-09-10T19:04:22.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:22.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1185087263' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi119-79837-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:22.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:22.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi119-79837-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:22.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:23.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:04:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:04:22] "GET /metrics HTTP/1.1" 200 261079 "" "Prometheus/2.43.0" 2024-09-10T19:04:23.270 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-10T19:04:23.270 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-10T19:04:23.270 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-10T19:04:23.270 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-10T19:04:23.270 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-10T19:04:23.270 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: notify 2024-09-10T19:04:23.270 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1047 ms) 2024-09-10T19:04:23.270 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-10T19:04:23.271 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (3 ms) 2024-09-10T19:04:23.271 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1050 ms total) 2024-09-10T19:04:23.271 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: 2024-09-10T19:04:23.271 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-10T19:04:23.271 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-10T19:04:23.271 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: notify 2024-09-10T19:04:23.271 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (43 ms) 2024-09-10T19:04:23.271 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-10T19:04:23.271 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: notify 2024-09-10T19:04:23.271 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3718 ms) 2024-09-10T19:04:23.271 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-10T19:04:23.271 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (21 ms) 2024-09-10T19:04:23.271 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-10T19:04:23.272 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (8 ms) 2024-09-10T19:04:23.272 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-10T19:04:23.272 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: handle_notify cookie 94514345401440 notify_id 343597383681 notifier_gid 25097 2024-09-10T19:04:23.272 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (6 ms) 2024-09-10T19:04:23.272 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-10T19:04:23.272 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: handle_notify cookie 94514345401440 notify_id 343597383682 notifier_gid 25097 2024-09-10T19:04:23.272 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (5 ms) 2024-09-10T19:04:23.272 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-10T19:04:23.272 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: handle_notify cookie 94514345396064 notify_id 343597383683 notifier_gid 25097 2024-09-10T19:04:23.272 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (5 ms) 2024-09-10T19:04:23.272 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-10T19:04:23.272 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: handle_notify cookie 94514345396064 notify_id 343597383684 notifier_gid 25097 2024-09-10T19:04:23.272 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (6 ms) 2024-09-10T19:04:23.273 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-10T19:04:23.273 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: handle_notify cookie 94514345396064 notify_id 343597383685 notifier_gid 25097 2024-09-10T19:04:23.273 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (8 ms) 2024-09-10T19:04:23.273 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-10T19:04:23.273 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: handle_notify cookie 94514345400416 notify_id 343597383686 notifier_gid 25097 2024-09-10T19:04:23.273 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (15 ms) 2024-09-10T19:04:23.273 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-10T19:04:23.273 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: trying... 2024-09-10T19:04:23.273 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: handle_notify cookie 94514345396064 notify_id 343597383687 notifier_gid 25097 2024-09-10T19:04:23.273 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: timed out 2024-09-10T19:04:23.273 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: flushing 2024-09-10T19:04:23.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:23 smithi194 ceph-mon[32464]: pgmap v84: 488 pgs: 128 unknown, 4 creating+peering, 356 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-09-10T19:04:23.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:23 smithi194 ceph-mon[32464]: 21.5 deep-scrub starts 2024-09-10T19:04:23.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:23 smithi194 ceph-mon[32464]: 21.5 deep-scrub ok 2024-09-10T19:04:23.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:23 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:23 smithi194 ceph-mon[32464]: pool 'PoolQuotaPP_smithi119-79499-3' is full (reached quota's max_bytes: 4 KiB) 2024-09-10T19:04:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:23 smithi194 ceph-mon[32464]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T19:04:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi119-79960-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]': finished 2024-09-10T19:04:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi119-79837-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:23 smithi194 ceph-mon[32464]: osdmap e82: 8 total, 8 up, 8 in 2024-09-10T19:04:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2175699846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2012933337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi119-79477-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:23.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[29577]: pgmap v84: 488 pgs: 128 unknown, 4 creating+peering, 356 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-09-10T19:04:23.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[29577]: 21.5 deep-scrub starts 2024-09-10T19:04:23.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[29577]: 21.5 deep-scrub ok 2024-09-10T19:04:23.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:23.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[29577]: pool 'PoolQuotaPP_smithi119-79499-3' is full (reached quota's max_bytes: 4 KiB) 2024-09-10T19:04:23.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[29577]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T19:04:23.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi119-79960-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]': finished 2024-09-10T19:04:23.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:23.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi119-79837-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:23.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[29577]: osdmap e82: 8 total, 8 up, 8 in 2024-09-10T19:04:23.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2175699846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2012933337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi119-79477-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:23.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[40674]: pgmap v84: 488 pgs: 128 unknown, 4 creating+peering, 356 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-09-10T19:04:23.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[40674]: 21.5 deep-scrub starts 2024-09-10T19:04:23.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[40674]: 21.5 deep-scrub ok 2024-09-10T19:04:23.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:23.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[40674]: pool 'PoolQuotaPP_smithi119-79499-3' is full (reached quota's max_bytes: 4 KiB) 2024-09-10T19:04:23.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[40674]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T19:04:23.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi119-79960-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]': finished 2024-09-10T19:04:23.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:23.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi119-79837-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:23.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[40674]: osdmap e82: 8 total, 8 up, 8 in 2024-09-10T19:04:23.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2175699846' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2012933337' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi119-79477-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:23.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:24.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:24 smithi194 ceph-mon[32464]: pgmap v87: 528 pgs: 72 creating+peering, 456 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-09-10T19:04:24.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:24.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2012933337' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi119-79477-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:24.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:24 smithi194 ceph-mon[32464]: osdmap e83: 8 total, 8 up, 8 in 2024-09-10T19:04:24.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:24.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2954281853' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:24.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:24.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:24.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:24.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[29577]: pgmap v87: 528 pgs: 72 creating+peering, 456 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-09-10T19:04:24.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:24.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2012933337' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi119-79477-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:24.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[29577]: osdmap e83: 8 total, 8 up, 8 in 2024-09-10T19:04:24.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:24.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2954281853' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:24.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:24.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:24.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:24.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[40674]: pgmap v87: 528 pgs: 72 creating+peering, 456 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-09-10T19:04:24.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:24.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2012933337' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi119-79477-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:24.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[40674]: osdmap e83: 8 total, 8 up, 8 in 2024-09-10T19:04:24.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:24.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2954281853' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:24.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:24.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:24.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[29577]: osdmap e84: 8 total, 8 up, 8 in 2024-09-10T19:04:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:25.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:25.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:25.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:25.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]': finished 2024-09-10T19:04:25.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:25.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[29577]: osdmap e85: 8 total, 8 up, 8 in 2024-09-10T19:04:25.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:25.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:25.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[40674]: osdmap e84: 8 total, 8 up, 8 in 2024-09-10T19:04:25.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:25.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:25.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:25.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:25.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:25.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]': finished 2024-09-10T19:04:25.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:25.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:25 smithi119 ceph-mon[40674]: osdmap e85: 8 total, 8 up, 8 in 2024-09-10T19:04:26.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:26.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:26.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:25 smithi194 ceph-mon[32464]: osdmap e84: 8 total, 8 up, 8 in 2024-09-10T19:04:26.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:25 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:26.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:26.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:25 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:26.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:25 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:26.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:26.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi119-79960-12"}]': finished 2024-09-10T19:04:26.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:26.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:25 smithi194 ceph-mon[32464]: osdmap e85: 8 total, 8 up, 8 in 2024-09-10T19:04:26.229 INFO:tasks.workunit.client.0.smithi119.stdout: api_wat api_watch_notify: Running main() from gmock_main.cc 2024-09-10T19:04:26.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-10T19:04:26.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-10T19:04:26.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-10T19:04:26.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-10T19:04:26.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: watch_notify_test_cb 2024-09-10T19:04:26.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (209 ms) 2024-09-10T19:04:26.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-10T19:04:26.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-10T19:04:26.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94071116453520 err -107 2024-09-10T19:04:26.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1057 ms) 2024-09-10T19:04:26.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-10T19:04:26.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94071116459840 err -107 2024-09-10T19:04:26.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-10T19:04:26.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (11 ms) 2024-09-10T19:04:26.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-10T19:04:26.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: watch_notify2_test_cb from 24692 notify_id 296352743424 cookie 94071116474048 2024-09-10T19:04:26.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (8 ms) 2024-09-10T19:04:26.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-10T19:04:26.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: watch_notify2_test_cb from 24692 notify_id 296352743425 cookie 94071116492160 2024-09-10T19:04:26.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (6 ms) 2024-09-10T19:04:26.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-10T19:04:26.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: watch_notify2_test_cb from 24692 notify_id 296352743424 cookie 94071116498480 2024-09-10T19:04:26.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (7 ms) 2024-09-10T19:04:26.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: watch_notify2_test_cb from 24692 notify_id 296352743426 cookie 94071116498480 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: watch_notify2_test_cb from 24692 notify_id 296352743426 cookie 94071116525520 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (5 ms) 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: watch_notify2_test_cb from 24692 notify_id 296352743425 cookie 94071116498480 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: watch_notify2_test_cb from 24692 notify_id 300647710722 cookie 94071116498480 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3022 ms) 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94071116498480 err -107 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: watch_notify2_test_cb from 24692 notify_id 330712481794 cookie 94071116498480 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5040 ms) 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-10T19:04:26.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94071116510752 err -107 2024-09-10T19:04:26.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1005 ms) 2024-09-10T19:04:26.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (10370 ms total) 2024-09-10T19:04:26.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: 2024-09-10T19:04:26.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-10T19:04:26.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-10T19:04:26.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: watch_notify_test_cb 2024-09-10T19:04:26.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1002 ms) 2024-09-10T19:04:26.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1002 ms total) 2024-09-10T19:04:26.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: 2024-09-10T19:04:26.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-10T19:04:26.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (24816 ms total) 2024-09-10T19:04:26.234 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-10T19:04:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: pgmap v90: 520 pgs: 64 unknown, 456 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 7.0 KiB/s wr, 7 op/s 2024-09-10T19:04:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3838676432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi119-79477-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3362444923' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-7"}]: dispatch 2024-09-10T19:04:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-7"}]: dispatch 2024-09-10T19:04:26.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi119-79477-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:26.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:26.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:26.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]': finished 2024-09-10T19:04:26.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-7"}]': finished 2024-09-10T19:04:26.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi119-79477-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:26.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:26.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: osdmap e86: 8 total, 8 up, 8 in 2024-09-10T19:04:26.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:26.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:26.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: pgmap v90: 520 pgs: 64 unknown, 456 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 7.0 KiB/s wr, 7 op/s 2024-09-10T19:04:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3838676432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi119-79477-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3362444923' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-7"}]: dispatch 2024-09-10T19:04:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-7"}]: dispatch 2024-09-10T19:04:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi119-79477-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]': finished 2024-09-10T19:04:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-7"}]': finished 2024-09-10T19:04:26.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi119-79477-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:26.821 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:26.821 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: osdmap e86: 8 total, 8 up, 8 in 2024-09-10T19:04:26.821 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:26.821 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:27.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: pgmap v90: 520 pgs: 64 unknown, 456 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 7.0 KiB/s wr, 7 op/s 2024-09-10T19:04:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3436746471' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3838676432' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi119-79477-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3362444923' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-7"}]: dispatch 2024-09-10T19:04:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]: dispatch 2024-09-10T19:04:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-7"}]: dispatch 2024-09-10T19:04:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi119-79477-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi119-79960-12"}]': finished 2024-09-10T19:04:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-7"}]': finished 2024-09-10T19:04:27.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi119-79477-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:27.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-80224-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:27.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: osdmap e86: 8 total, 8 up, 8 in 2024-09-10T19:04:27.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:27.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:27.222 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-10T19:04:27.222 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-10T19:04:27.222 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-10T19:04:27.222 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-10T19:04:27.222 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-10T19:04:27.223 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-10T19:04:27.223 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-10T19:04:27.223 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: 2024-09-10T19:04:27.223 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-10T19:04:27.223 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-10T19:04:27.223 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (8160 ms) 2024-09-10T19:04:27.223 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-10T19:04:27.223 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3273 ms) 2024-09-10T19:04:27.223 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-10T19:04:27.223 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (2515 ms) 2024-09-10T19:04:27.223 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-10T19:04:27.223 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2676 ms) 2024-09-10T19:04:27.223 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-10T19:04:27.223 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3351 ms) 2024-09-10T19:04:27.224 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-10T19:04:27.224 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (2756 ms) 2024-09-10T19:04:27.224 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-10T19:04:27.224 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (3007 ms) 2024-09-10T19:04:27.224 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (25739 ms total) 2024-09-10T19:04:27.224 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: 2024-09-10T19:04:27.224 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-10T19:04:27.224 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (25739 ms total) 2024-09-10T19:04:27.224 INFO:tasks.workunit.client.0.smithi119.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-10T19:04:28.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:04:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:04:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:27 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:27 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:04:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:27 smithi194 ceph-mon[32464]: osdmap e87: 8 total, 8 up, 8 in 2024-09-10T19:04:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7"}]: dispatch 2024-09-10T19:04:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:27 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7"}]: dispatch 2024-09-10T19:04:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:28.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:28.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:27 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:28.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:27 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:04:28.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:27 smithi119 ceph-mon[29577]: osdmap e87: 8 total, 8 up, 8 in 2024-09-10T19:04:28.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7"}]: dispatch 2024-09-10T19:04:28.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:27 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7"}]: dispatch 2024-09-10T19:04:28.320 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:28.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:28.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:27 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:28.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:27 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:04:28.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:27 smithi119 ceph-mon[40674]: osdmap e87: 8 total, 8 up, 8 in 2024-09-10T19:04:28.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7"}]: dispatch 2024-09-10T19:04:28.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:27 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7"}]: dispatch 2024-09-10T19:04:28.321 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:29.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:28 smithi194 ceph-mon[32464]: pgmap v93: 584 pgs: 128 unknown, 456 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:29.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:28 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:29.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:28 smithi194 ceph-mon[32464]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T19:04:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7"}]': finished 2024-09-10T19:04:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:28 smithi194 ceph-mon[32464]: osdmap e88: 8 total, 8 up, 8 in 2024-09-10T19:04:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3221283048' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi119-79477-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi119-79477-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/469323501' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[29577]: pgmap v93: 584 pgs: 128 unknown, 456 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[29577]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T19:04:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7"}]': finished 2024-09-10T19:04:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[29577]: osdmap e88: 8 total, 8 up, 8 in 2024-09-10T19:04:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3221283048' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi119-79477-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi119-79477-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/469323501' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:29.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[40674]: pgmap v93: 584 pgs: 128 unknown, 456 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[40674]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T19:04:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-7"}]': finished 2024-09-10T19:04:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[40674]: osdmap e88: 8 total, 8 up, 8 in 2024-09-10T19:04:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3221283048' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi119-79477-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi119-79477-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/469323501' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:30.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:04:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:04:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:04:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:30 smithi194 ceph-mon[32464]: pgmap v96: 520 pgs: 64 unknown, 456 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-10T19:04:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:30 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi119-79477-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:30 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:30 smithi194 ceph-mon[32464]: osdmap e89: 8 total, 8 up, 8 in 2024-09-10T19:04:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:30 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:30 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:30.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:30 smithi119 ceph-mon[29577]: pgmap v96: 520 pgs: 64 unknown, 456 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-10T19:04:30.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:30 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi119-79477-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:30.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:30 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:30.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:30 smithi119 ceph-mon[29577]: osdmap e89: 8 total, 8 up, 8 in 2024-09-10T19:04:30.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:30 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:30.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:30 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:30.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:30 smithi119 ceph-mon[40674]: pgmap v96: 520 pgs: 64 unknown, 456 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s 2024-09-10T19:04:30.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:30 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi119-79477-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:30.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:30 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:30.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:30 smithi119 ceph-mon[40674]: osdmap e89: 8 total, 8 up, 8 in 2024-09-10T19:04:30.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:30 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:30.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:30 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:31.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:31.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:31.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:31 smithi194 ceph-mon[32464]: osdmap e90: 8 total, 8 up, 8 in 2024-09-10T19:04:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3494575704' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[40674]: osdmap e90: 8 total, 8 up, 8 in 2024-09-10T19:04:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3494575704' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:31.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:31.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:31.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:31.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[29577]: osdmap e90: 8 total, 8 up, 8 in 2024-09-10T19:04:31.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3494575704' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:31.928 INFO:tasks.workunit.client.0.smithi119.stdout:vice: foo: 16 portals active (1 hosts, 3 zones) 2024-09-10T19:04:31.929 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-10T19:04:31.929 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-09-10T19:04:31.929 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:31.929 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: data: 2024-09-10T19:04:31.929 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: pools: 27 pools, 804 pgs 2024-09-10T19:04:31.929 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: objects: 220 objects, 584 KiB 2024-09-10T19:04:31.929 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: usage: 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:04:31.929 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: pgs: 3.980% pgs unknown 2024-09-10T19:04:31.929 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 59.701% pgs not active 2024-09-10T19:04:31.929 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 480 creating+peering 2024-09-10T19:04:31.929 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 292 active+clean 2024-09-10T19:04:31.929 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 32 unknown 2024-09-10T19:04:31.929 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:31.929 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: io: 2024-09-10T19:04:31.930 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: client: 2.0 KiB/s rd, 12 KiB/s wr, 3 op/s rd, 20 op/s wr 2024-09-10T19:04:31.930 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:31.930 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:31.930 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: [ OK ] LibRadosService.StatusFormat (4941 ms) 2024-09-10T19:04:31.930 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-10T19:04:31.930 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: [ OK ] LibRadosService.Status (20411 ms) 2024-09-10T19:04:31.930 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: [----------] 4 tests from LibRadosService (30417 ms total) 2024-09-10T19:04:31.930 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: 2024-09-10T19:04:31.930 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: [----------] Global test environment tear-down 2024-09-10T19:04:31.930 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: [==========] 4 tests from 1 test suite ran. (30417 ms total) 2024-09-10T19:04:31.930 INFO:tasks.workunit.client.0.smithi119.stdout: api_service: [ PASSED ] 4 tests. 2024-09-10T19:04:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: pgmap v99: 488 pgs: 128 unknown, 360 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-09-10T19:04:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: osdmap e91: 8 total, 8 up, 8 in 2024-09-10T19:04:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi119-79823-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3477213746' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi119-79477-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi119-79823-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi119-79477-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:32.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:32.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:32.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:32.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:32.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:32.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi119-79477-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:32.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-9"}]: dispatch 2024-09-10T19:04:32.819 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:04:32] "GET /metrics HTTP/1.1" 200 249480 "" "Prometheus/2.43.0" 2024-09-10T19:04:32.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: pgmap v99: 488 pgs: 128 unknown, 360 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-09-10T19:04:32.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:32.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:32.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:32.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: osdmap e91: 8 total, 8 up, 8 in 2024-09-10T19:04:32.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi119-79823-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:32.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3477213746' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi119-79477-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:32.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi119-79823-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:32.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi119-79477-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:32.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:32.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:32.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:32.821 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:32.821 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:32.821 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi119-79477-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:32.821 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-9"}]: dispatch 2024-09-10T19:04:33.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: pgmap v99: 488 pgs: 128 unknown, 360 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 2 op/s 2024-09-10T19:04:33.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:33.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:33.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: osdmap e91: 8 total, 8 up, 8 in 2024-09-10T19:04:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi119-79823-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3477213746' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi119-79477-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi119-79823-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi119-79477-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi119-79477-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-9"}]: dispatch 2024-09-10T19:04:33.059 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-10T19:04:33.059 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-10T19:04:33.059 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-10T19:04:33.059 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-10T19:04:33.059 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5835 ms) 2024-09-10T19:04:33.059 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-10T19:04:33.059 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (28 ms) 2024-09-10T19:04:33.059 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-10T19:04:33.059 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20092 ms) 2024-09-10T19:04:33.059 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-10T19:04:33.060 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: attempt 0 of 20 2024-09-10T19:04:33.060 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5622 ms) 2024-09-10T19:04:33.060 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (31577 ms total) 2024-09-10T19:04:33.060 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: 2024-09-10T19:04:33.060 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-10T19:04:33.060 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (31577 ms total) 2024-09-10T19:04:33.060 INFO:tasks.workunit.client.0.smithi119.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-10T19:04:34.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4"}]: dispatch 2024-09-10T19:04:34.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: osdmap e92: 8 total, 8 up, 8 in 2024-09-10T19:04:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4"}]: dispatch 2024-09-10T19:04:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-9"}]: dispatch 2024-09-10T19:04:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4001612417' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T19:04:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi119-79823-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]': finished 2024-09-10T19:04:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4"}]': finished 2024-09-10T19:04:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-9"}]': finished 2024-09-10T19:04:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: osdmap e93: 8 total, 8 up, 8 in 2024-09-10T19:04:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]: dispatch 2024-09-10T19:04:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-9", "mode": "writeback"}]: dispatch 2024-09-10T19:04:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]: dispatch 2024-09-10T19:04:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-9", "mode": "writeback"}]: dispatch 2024-09-10T19:04:34.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/111964957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:34.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:34.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/796493948' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:34.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4"}]: dispatch 2024-09-10T19:04:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: osdmap e92: 8 total, 8 up, 8 in 2024-09-10T19:04:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4"}]: dispatch 2024-09-10T19:04:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-9"}]: dispatch 2024-09-10T19:04:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4001612417' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T19:04:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi119-79823-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]': finished 2024-09-10T19:04:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4"}]': finished 2024-09-10T19:04:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-9"}]': finished 2024-09-10T19:04:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: osdmap e93: 8 total, 8 up, 8 in 2024-09-10T19:04:34.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]: dispatch 2024-09-10T19:04:34.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-9", "mode": "writeback"}]: dispatch 2024-09-10T19:04:34.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]: dispatch 2024-09-10T19:04:34.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-9", "mode": "writeback"}]: dispatch 2024-09-10T19:04:34.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/111964957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:34.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:34.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/796493948' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:34.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:34.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4"}]: dispatch 2024-09-10T19:04:34.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:34.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: osdmap e92: 8 total, 8 up, 8 in 2024-09-10T19:04:34.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4"}]: dispatch 2024-09-10T19:04:34.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-9"}]: dispatch 2024-09-10T19:04:34.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:34.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4001612417' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T19:04:34.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi119-79823-9", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]': finished 2024-09-10T19:04:34.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4"}]': finished 2024-09-10T19:04:34.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-9"}]': finished 2024-09-10T19:04:34.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: osdmap e93: 8 total, 8 up, 8 in 2024-09-10T19:04:34.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]: dispatch 2024-09-10T19:04:34.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-9", "mode": "writeback"}]: dispatch 2024-09-10T19:04:34.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]: dispatch 2024-09-10T19:04:34.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-9", "mode": "writeback"}]: dispatch 2024-09-10T19:04:34.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/111964957' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:34.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:34.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/796493948' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:34.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:35.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T19:04:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: pgmap v102: 424 pgs: 424 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-09-10T19:04:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]': finished 2024-09-10T19:04:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-9", "mode": "writeback"}]': finished 2024-09-10T19:04:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: osdmap e94: 8 total, 8 up, 8 in 2024-09-10T19:04:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:04:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:04:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1798901234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi119-79477-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi119-79477-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:35.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T19:04:35.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:35.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T19:04:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T19:04:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: pgmap v102: 424 pgs: 424 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-09-10T19:04:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]': finished 2024-09-10T19:04:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-9", "mode": "writeback"}]': finished 2024-09-10T19:04:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: osdmap e94: 8 total, 8 up, 8 in 2024-09-10T19:04:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:04:35.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:04:35.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1798901234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi119-79477-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:35.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi119-79477-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:35.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:35.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:35.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T19:04:35.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:35.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T19:04:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-09-10T19:04:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: pgmap v102: 424 pgs: 424 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-09-10T19:04:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-4", "tierpool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6"}]': finished 2024-09-10T19:04:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-9", "mode": "writeback"}]': finished 2024-09-10T19:04:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi119-79499-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: osdmap e94: 8 total, 8 up, 8 in 2024-09-10T19:04:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2280773366' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:04:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:04:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1798901234' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi119-79477-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi119-79477-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:35.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:35.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T19:04:35.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:35.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T19:04:36.338 INFO:tasks.workunit.client.0.smithi119.stdout:ch_notify_pp: flushed 2024-09-10T19:04:36.338 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3005 ms) 2024-09-10T19:04:36.338 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-10T19:04:36.338 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: trying... 2024-09-10T19:04:36.338 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: handle_notify cookie 94514345396064 notify_id 356482285577 notifier_gid 25097 2024-09-10T19:04:36.338 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: timed out 2024-09-10T19:04:36.339 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: flushing 2024-09-10T19:04:36.339 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: flushed 2024-09-10T19:04:36.339 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3021 ms) 2024-09-10T19:04:36.339 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-10T19:04:36.339 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: List watches 2024-09-10T19:04:36.339 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: notify2 2024-09-10T19:04:36.339 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: handle_notify cookie 94514345396064 notify_id 369367187460 notifier_gid 25097 2024-09-10T19:04:36.339 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: notify2 done 2024-09-10T19:04:36.339 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: watch_check 2024-09-10T19:04:36.339 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: unwatch2 2024-09-10T19:04:36.339 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: flushing 2024-09-10T19:04:36.339 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: done 2024-09-10T19:04:36.339 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3023 ms) 2024-09-10T19:04:36.339 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-10T19:04:36.340 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: List watches 2024-09-10T19:04:36.340 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: notify2 2024-09-10T19:04:36.340 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: handle_notify cookie 94514345396064 notify_id 382252089354 notifier_gid 25097 2024-09-10T19:04:36.340 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: notify2 done 2024-09-10T19:04:36.340 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: watch_check 2024-09-10T19:04:36.340 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: unwatch2 2024-09-10T19:04:36.340 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: flushing 2024-09-10T19:04:36.340 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: done 2024-09-10T19:04:36.340 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3099 ms) 2024-09-10T19:04:36.340 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (15983 ms total) 2024-09-10T19:04:36.340 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: 2024-09-10T19:04:36.340 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-10T19:04:36.340 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (34893 ms total) 2024-09-10T19:04:36.341 INFO:tasks.workunit.client.0.smithi119.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-10T19:04:36.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:36 smithi194 ceph-mon[32464]: pgmap v105: 496 pgs: 104 unknown, 392 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-09-10T19:04:36.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:04:36.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi119-79477-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:36.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:04:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]': finished 2024-09-10T19:04:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:36 smithi194 ceph-mon[32464]: osdmap e95: 8 total, 8 up, 8 in 2024-09-10T19:04:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9"}]: dispatch 2024-09-10T19:04:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9"}]: dispatch 2024-09-10T19:04:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T19:04:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T19:04:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[29577]: pgmap v105: 496 pgs: 104 unknown, 392 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-09-10T19:04:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:04:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi119-79477-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:04:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]': finished 2024-09-10T19:04:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[29577]: osdmap e95: 8 total, 8 up, 8 in 2024-09-10T19:04:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9"}]: dispatch 2024-09-10T19:04:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9"}]: dispatch 2024-09-10T19:04:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T19:04:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T19:04:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[40674]: pgmap v105: 496 pgs: 104 unknown, 392 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2024-09-10T19:04:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "pool2": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi119-80068-6", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:04:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi119-79477-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:04:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]': finished 2024-09-10T19:04:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[40674]: osdmap e95: 8 total, 8 up, 8 in 2024-09-10T19:04:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9"}]: dispatch 2024-09-10T19:04:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9"}]: dispatch 2024-09-10T19:04:36.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:36.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T19:04:36.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T19:04:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9"}]': finished 2024-09-10T19:04:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]': finished 2024-09-10T19:04:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[29577]: osdmap e96: 8 total, 8 up, 8 in 2024-09-10T19:04:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3168073269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi119-79499-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3168073269' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi119-79499-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[29577]: osdmap e97: 8 total, 8 up, 8 in 2024-09-10T19:04:37.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:37.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9"}]': finished 2024-09-10T19:04:37.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]': finished 2024-09-10T19:04:37.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[40674]: osdmap e96: 8 total, 8 up, 8 in 2024-09-10T19:04:37.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:37.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3168073269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi119-79499-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:37.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:37.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3168073269' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi119-79499-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:37.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:37 smithi119 ceph-mon[40674]: osdmap e97: 8 total, 8 up, 8 in 2024-09-10T19:04:38.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:37 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:38.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-9"}]': finished 2024-09-10T19:04:38.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]': finished 2024-09-10T19:04:38.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:37 smithi194 ceph-mon[32464]: osdmap e96: 8 total, 8 up, 8 in 2024-09-10T19:04:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3168073269' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi119-79499-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:37 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3168073269' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi119-79499-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:37 smithi194 ceph-mon[32464]: osdmap e97: 8 total, 8 up, 8 in 2024-09-10T19:04:38.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:04:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:04:39.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:38 smithi194 ceph-mon[32464]: pgmap v108: 428 pgs: 72 unknown, 356 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2814988200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi119-79477-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi119-79477-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi119-79477-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:38 smithi194 ceph-mon[32464]: osdmap e98: 8 total, 8 up, 8 in 2024-09-10T19:04:39.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3392458182' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[29577]: pgmap v108: 428 pgs: 72 unknown, 356 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2814988200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi119-79477-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi119-79477-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi119-79477-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[29577]: osdmap e98: 8 total, 8 up, 8 in 2024-09-10T19:04:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3392458182' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[40674]: pgmap v108: 428 pgs: 72 unknown, 356 active+clean; 145 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:39.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2814988200' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi119-79477-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:39.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi119-79477-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi119-79477-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:39.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[40674]: osdmap e98: 8 total, 8 up, 8 in 2024-09-10T19:04:39.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3392458182' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:39.210 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-10T19:04:39.210 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [----------] Global test environment set-up. 2024-09-10T19:04:39.210 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-10T19:04:39.210 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-10T19:04:39.210 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-10T19:04:39.210 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-10T19:04:39.210 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: 2024-09-10T19:04:39.210 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-10T19:04:39.210 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-10T19:04:39.210 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-10T19:04:39.211 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: 2024-09-10T19:04:01.381+0000 7efc49694600 -1 failed for service _ceph-mon._tcp 2024-09-10T19:04:39.211 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: 2024-09-10T19:04:01.381+0000 7efc49694600 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-10T19:04:39.211 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (68 ms) 2024-09-10T19:04:39.211 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-10T19:04:39.211 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5012 ms) 2024-09-10T19:04:39.211 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5080 ms total) 2024-09-10T19:04:39.211 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: 2024-09-10T19:04:39.211 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-10T19:04:39.211 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-10T19:04:39.211 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: started 0x55748f9222d0 2024-09-10T19:04:39.211 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: started 0x55748f926ad0 2024-09-10T19:04:39.211 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: started 2 aios 2024-09-10T19:04:39.211 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: waiting 0x55748f9222d0 2024-09-10T19:04:39.211 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: waiting 0x55748f926ad0 2024-09-10T19:04:39.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: done. 2024-09-10T19:04:39.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (7280 ms) 2024-09-10T19:04:39.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (7280 ms total) 2024-09-10T19:04:39.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: 2024-09-10T19:04:39.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-10T19:04:39.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-10T19:04:39.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-10T19:04:39.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-10T19:04:39.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ OK ] LibRadosMisc.Exec (64 ms) 2024-09-10T19:04:39.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-10T19:04:39.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (10 ms) 2024-09-10T19:04:39.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-10T19:04:39.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (5 ms) 2024-09-10T19:04:39.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-10T19:04:39.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4595 ms) 2024-09-10T19:04:39.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-10T19:04:39.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-10T19:04:39.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-10T19:04:39.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-10T19:04:39.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-10T19:04:39.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (18816 ms) 2024-09-10T19:04:39.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [----------] 8 tests from LibRadosMisc (23490 ms total) 2024-09-10T19:04:39.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: 2024-09-10T19:04:39.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [----------] Global test environment tear-down 2024-09-10T19:04:39.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (37875 ms total) 2024-09-10T19:04:39.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-10T19:04:40.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:39 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:40.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:39 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:40.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:40.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:40.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:39 smithi194 ceph-mon[32464]: osdmap e99: 8 total, 8 up, 8 in 2024-09-10T19:04:40.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:39 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/666284244' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi119-79499-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:40.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi119-79499-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:40.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:39 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:40.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:40.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:40.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:40.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:40.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[29577]: osdmap e99: 8 total, 8 up, 8 in 2024-09-10T19:04:40.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/666284244' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi119-79499-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:40.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi119-79499-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:40.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:40.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:40.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:40.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79774-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:40.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:40.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[40674]: osdmap e99: 8 total, 8 up, 8 in 2024-09-10T19:04:40.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/666284244' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi119-79499-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:40.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi119-79499-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:40.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:39 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:40.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:04:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:04:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:04:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:40 smithi194 ceph-mon[32464]: pgmap v111: 428 pgs: 32 creating+peering, 64 unknown, 332 active+clean; 146 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:40 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi119-79499-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:40 smithi194 ceph-mon[32464]: osdmap e100: 8 total, 8 up, 8 in 2024-09-10T19:04:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3392458182' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache", "force_nonempty":""}]: dispatch 2024-09-10T19:04:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2711115129' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi119-79477-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:40 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache", "force_nonempty":""}]: dispatch 2024-09-10T19:04:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:40 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi119-79477-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:40 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[29577]: pgmap v111: 428 pgs: 32 creating+peering, 64 unknown, 332 active+clean; 146 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi119-79499-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[29577]: osdmap e100: 8 total, 8 up, 8 in 2024-09-10T19:04:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3392458182' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache", "force_nonempty":""}]: dispatch 2024-09-10T19:04:41.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2711115129' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi119-79477-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:41.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache", "force_nonempty":""}]: dispatch 2024-09-10T19:04:41.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi119-79477-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:41.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:41.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:41.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:41.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[40674]: pgmap v111: 428 pgs: 32 creating+peering, 64 unknown, 332 active+clean; 146 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:41.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi119-79499-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:41.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[40674]: osdmap e100: 8 total, 8 up, 8 in 2024-09-10T19:04:41.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3392458182' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache", "force_nonempty":""}]: dispatch 2024-09-10T19:04:41.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2711115129' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi119-79477-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:41.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache", "force_nonempty":""}]: dispatch 2024-09-10T19:04:41.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi119-79477-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:41.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:41.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:41.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.0"}]: dispatch 2024-09-10T19:04:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.0"}]: dispatch 2024-09-10T19:04:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.1"}]: dispatch 2024-09-10T19:04:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.1"}]: dispatch 2024-09-10T19:04:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.2"}]: dispatch 2024-09-10T19:04:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.2"}]: dispatch 2024-09-10T19:04:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.3"}]: dispatch 2024-09-10T19:04:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.3"}]: dispatch 2024-09-10T19:04:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.4"}]: dispatch 2024-09-10T19:04:42.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.4"}]: dispatch 2024-09-10T19:04:42.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.5"}]: dispatch 2024-09-10T19:04:42.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.5"}]: dispatch 2024-09-10T19:04:42.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.6"}]: dispatch 2024-09-10T19:04:42.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.6"}]: dispatch 2024-09-10T19:04:42.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.7"}]: dispatch 2024-09-10T19:04:42.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.7"}]: dispatch 2024-09-10T19:04:42.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.8"}]: dispatch 2024-09-10T19:04:42.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.8"}]: dispatch 2024-09-10T19:04:42.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.9"}]: dispatch 2024-09-10T19:04:42.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.9"}]: dispatch 2024-09-10T19:04:42.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:04:42.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: 21.8 deep-scrub starts 2024-09-10T19:04:42.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: 21.8 deep-scrub ok 2024-09-10T19:04:42.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: 21.0 deep-scrub starts 2024-09-10T19:04:42.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: 21.0 deep-scrub ok 2024-09-10T19:04:42.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: pgmap v114: 460 pgs: 160 unknown, 300 active+clean; 146 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 256 KiB/s wr, 1 op/s 2024-09-10T19:04:42.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache", "force_nonempty":""}]': finished 2024-09-10T19:04:42.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi119-79477-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:42.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:42.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: osdmap e101: 8 total, 8 up, 8 in 2024-09-10T19:04:42.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-11"}]: dispatch 2024-09-10T19:04:42.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-11"}]: dispatch 2024-09-10T19:04:42.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3392458182' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79774-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:42.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79774-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:42.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:42.022 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:04:42.022 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:04:42.022 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:41 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:04:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.0"}]: dispatch 2024-09-10T19:04:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.0"}]: dispatch 2024-09-10T19:04:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.1"}]: dispatch 2024-09-10T19:04:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.1"}]: dispatch 2024-09-10T19:04:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.2"}]: dispatch 2024-09-10T19:04:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.2"}]: dispatch 2024-09-10T19:04:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.3"}]: dispatch 2024-09-10T19:04:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.3"}]: dispatch 2024-09-10T19:04:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.4"}]: dispatch 2024-09-10T19:04:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.4"}]: dispatch 2024-09-10T19:04:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.5"}]: dispatch 2024-09-10T19:04:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.5"}]: dispatch 2024-09-10T19:04:42.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.6"}]: dispatch 2024-09-10T19:04:42.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.6"}]: dispatch 2024-09-10T19:04:42.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.7"}]: dispatch 2024-09-10T19:04:42.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.7"}]: dispatch 2024-09-10T19:04:42.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.8"}]: dispatch 2024-09-10T19:04:42.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.8"}]: dispatch 2024-09-10T19:04:42.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.9"}]: dispatch 2024-09-10T19:04:42.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.9"}]: dispatch 2024-09-10T19:04:42.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:04:42.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: 21.8 deep-scrub starts 2024-09-10T19:04:42.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: 21.8 deep-scrub ok 2024-09-10T19:04:42.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: 21.0 deep-scrub starts 2024-09-10T19:04:42.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: 21.0 deep-scrub ok 2024-09-10T19:04:42.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: pgmap v114: 460 pgs: 160 unknown, 300 active+clean; 146 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 256 KiB/s wr, 1 op/s 2024-09-10T19:04:42.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache", "force_nonempty":""}]': finished 2024-09-10T19:04:42.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi119-79477-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:42.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.0"}]: dispatch 2024-09-10T19:04:42.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.0"}]: dispatch 2024-09-10T19:04:42.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.1"}]: dispatch 2024-09-10T19:04:42.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.1"}]: dispatch 2024-09-10T19:04:42.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.2"}]: dispatch 2024-09-10T19:04:42.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.2"}]: dispatch 2024-09-10T19:04:42.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.3"}]: dispatch 2024-09-10T19:04:42.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.3"}]: dispatch 2024-09-10T19:04:42.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.4"}]: dispatch 2024-09-10T19:04:42.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.4"}]: dispatch 2024-09-10T19:04:42.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.5"}]: dispatch 2024-09-10T19:04:42.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.5"}]: dispatch 2024-09-10T19:04:42.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.6"}]: dispatch 2024-09-10T19:04:42.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.6"}]: dispatch 2024-09-10T19:04:42.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.7"}]: dispatch 2024-09-10T19:04:42.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.7"}]: dispatch 2024-09-10T19:04:42.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.8"}]: dispatch 2024-09-10T19:04:42.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.8"}]: dispatch 2024-09-10T19:04:42.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.9"}]: dispatch 2024-09-10T19:04:42.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "21.9"}]: dispatch 2024-09-10T19:04:42.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:04:42.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: 21.8 deep-scrub starts 2024-09-10T19:04:42.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: 21.8 deep-scrub ok 2024-09-10T19:04:42.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: 21.0 deep-scrub starts 2024-09-10T19:04:42.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: 21.0 deep-scrub ok 2024-09-10T19:04:42.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: pgmap v114: 460 pgs: 160 unknown, 300 active+clean; 146 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 767 B/s rd, 256 KiB/s wr, 1 op/s 2024-09-10T19:04:42.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache", "force_nonempty":""}]': finished 2024-09-10T19:04:42.071 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi119-79477-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:42.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:42.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: osdmap e101: 8 total, 8 up, 8 in 2024-09-10T19:04:42.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-11"}]: dispatch 2024-09-10T19:04:42.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-11"}]: dispatch 2024-09-10T19:04:42.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3392458182' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79774-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:42.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79774-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:42.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:42.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:04:42.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:04:42.072 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:04:42.073 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:42.073 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: osdmap e101: 8 total, 8 up, 8 in 2024-09-10T19:04:42.073 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-11"}]: dispatch 2024-09-10T19:04:42.073 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-11"}]: dispatch 2024-09-10T19:04:42.073 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3392458182' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79774-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:42.073 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79774-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:42.073 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:42.073 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:04:42.073 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:04:42.073 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:41 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:04:43.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: 21.5 deep-scrub starts 2024-09-10T19:04:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: 21.5 deep-scrub ok 2024-09-10T19:04:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: 21.7 deep-scrub starts 2024-09-10T19:04:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: 21.7 deep-scrub ok 2024-09-10T19:04:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: 21.2 deep-scrub starts 2024-09-10T19:04:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: 21.2 deep-scrub ok 2024-09-10T19:04:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-11"}]': finished 2024-09-10T19:04:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79774-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: osdmap e102: 8 total, 8 up, 8 in 2024-09-10T19:04:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2711323430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi119-79499-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-11", "mode": "writeback"}]: dispatch 2024-09-10T19:04:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3392458182' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache"}]: dispatch 2024-09-10T19:04:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-11", "mode": "writeback"}]: dispatch 2024-09-10T19:04:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache"}]: dispatch 2024-09-10T19:04:43.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:43.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:04:42] "GET /metrics HTTP/1.1" 200 249480 "" "Prometheus/2.43.0" 2024-09-10T19:04:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: 21.5 deep-scrub starts 2024-09-10T19:04:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: 21.5 deep-scrub ok 2024-09-10T19:04:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: 21.7 deep-scrub starts 2024-09-10T19:04:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: 21.7 deep-scrub ok 2024-09-10T19:04:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: 21.2 deep-scrub starts 2024-09-10T19:04:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: 21.2 deep-scrub ok 2024-09-10T19:04:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-11"}]': finished 2024-09-10T19:04:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79774-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: osdmap e102: 8 total, 8 up, 8 in 2024-09-10T19:04:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2711323430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi119-79499-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-11", "mode": "writeback"}]: dispatch 2024-09-10T19:04:43.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3392458182' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache"}]: dispatch 2024-09-10T19:04:43.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-11", "mode": "writeback"}]: dispatch 2024-09-10T19:04:43.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache"}]: dispatch 2024-09-10T19:04:43.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: 21.5 deep-scrub starts 2024-09-10T19:04:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: 21.5 deep-scrub ok 2024-09-10T19:04:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: 21.7 deep-scrub starts 2024-09-10T19:04:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: 21.7 deep-scrub ok 2024-09-10T19:04:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: 21.2 deep-scrub starts 2024-09-10T19:04:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: 21.2 deep-scrub ok 2024-09-10T19:04:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:43.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-11"}]': finished 2024-09-10T19:04:43.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79774-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:43.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: osdmap e102: 8 total, 8 up, 8 in 2024-09-10T19:04:43.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2711323430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi119-79499-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:43.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-11", "mode": "writeback"}]: dispatch 2024-09-10T19:04:43.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3392458182' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache"}]: dispatch 2024-09-10T19:04:43.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-11", "mode": "writeback"}]: dispatch 2024-09-10T19:04:43.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache"}]: dispatch 2024-09-10T19:04:43.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:44.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:43 smithi194 ceph-mon[32464]: pgmap v117: 428 pgs: 32 creating+peering, 396 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 18 MiB/s wr, 4 op/s 2024-09-10T19:04:44.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:43 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:44.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2711323430' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi119-79499-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-11", "mode": "writeback"}]': finished 2024-09-10T19:04:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache"}]': finished 2024-09-10T19:04:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:43 smithi194 ceph-mon[32464]: osdmap e103: 8 total, 8 up, 8 in 2024-09-10T19:04:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4170187714' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi119-79477-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:44.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[29577]: pgmap v117: 428 pgs: 32 creating+peering, 396 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 18 MiB/s wr, 4 op/s 2024-09-10T19:04:44.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:44.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2711323430' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi119-79499-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:44.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-11", "mode": "writeback"}]': finished 2024-09-10T19:04:44.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache"}]': finished 2024-09-10T19:04:44.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[29577]: osdmap e103: 8 total, 8 up, 8 in 2024-09-10T19:04:44.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:44.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:44.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4170187714' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi119-79477-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:44.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:44.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[40674]: pgmap v117: 428 pgs: 32 creating+peering, 396 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 18 MiB/s wr, 4 op/s 2024-09-10T19:04:44.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:44.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2711323430' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi119-79499-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:44.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-11", "mode": "writeback"}]': finished 2024-09-10T19:04:44.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79774-10", "tierpool":"test-rados-api-smithi119-79774-10-cache"}]': finished 2024-09-10T19:04:44.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[40674]: osdmap e103: 8 total, 8 up, 8 in 2024-09-10T19:04:44.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:44.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:44.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4170187714' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi119-79477-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:44.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:45.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:44 smithi194 ceph-mon[32464]: 21.6 deep-scrub starts 2024-09-10T19:04:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:44 smithi194 ceph-mon[32464]: 21.6 deep-scrub ok 2024-09-10T19:04:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]': finished 2024-09-10T19:04:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4170187714' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi119-79477-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:44 smithi194 ceph-mon[32464]: osdmap e104: 8 total, 8 up, 8 in 2024-09-10T19:04:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:45.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[29577]: 21.6 deep-scrub starts 2024-09-10T19:04:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[29577]: 21.6 deep-scrub ok 2024-09-10T19:04:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]': finished 2024-09-10T19:04:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4170187714' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi119-79477-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[29577]: osdmap e104: 8 total, 8 up, 8 in 2024-09-10T19:04:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[40674]: 21.6 deep-scrub starts 2024-09-10T19:04:45.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[40674]: 21.6 deep-scrub ok 2024-09-10T19:04:45.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi119-79823-9"}]': finished 2024-09-10T19:04:45.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4170187714' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi119-79477-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:45.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[40674]: osdmap e104: 8 total, 8 up, 8 in 2024-09-10T19:04:45.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3619268231' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:45.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]: dispatch 2024-09-10T19:04:45.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:45.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:45.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:04:45.182 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: Running main() from gmock_main.cc 2024-09-10T19:04:45.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-10T19:04:45.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [----------] Global test environment set-up. 2024-09-10T19:04:45.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-10T19:04:45.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-10T19:04:45.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ OK ] LibRadosPools.PoolList (8293 ms) 2024-09-10T19:04:45.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-10T19:04:45.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3270 ms) 2024-09-10T19:04:45.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-10T19:04:45.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3522 ms) 2024-09-10T19:04:45.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-10T19:04:45.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2682 ms) 2024-09-10T19:04:45.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-10T19:04:45.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3111 ms) 2024-09-10T19:04:45.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-10T19:04:45.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5007 ms) 2024-09-10T19:04:45.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-10T19:04:45.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5307 ms) 2024-09-10T19:04:45.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-10T19:04:45.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4644 ms) 2024-09-10T19:04:45.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-10T19:04:45.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7994 ms) 2024-09-10T19:04:45.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [----------] 9 tests from LibRadosPools (43830 ms total) 2024-09-10T19:04:45.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: 2024-09-10T19:04:45.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [----------] Global test environment tear-down 2024-09-10T19:04:45.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (43830 ms total) 2024-09-10T19:04:45.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-10T19:04:46.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:46 smithi194 ceph-mon[32464]: pgmap v120: 388 pgs: 32 unknown, 356 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 4 op/s 2024-09-10T19:04:46.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]': finished 2024-09-10T19:04:46.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:04:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:46 smithi194 ceph-mon[32464]: osdmap e105: 8 total, 8 up, 8 in 2024-09-10T19:04:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2916522859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11"}]: dispatch 2024-09-10T19:04:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11"}]: dispatch 2024-09-10T19:04:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:46 smithi194 ceph-mon[32464]: 21.9 deep-scrub starts 2024-09-10T19:04:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:46 smithi194 ceph-mon[32464]: 21.9 deep-scrub ok 2024-09-10T19:04:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:46.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[40674]: pgmap v120: 388 pgs: 32 unknown, 356 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 4 op/s 2024-09-10T19:04:46.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]': finished 2024-09-10T19:04:46.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:04:46.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[40674]: osdmap e105: 8 total, 8 up, 8 in 2024-09-10T19:04:46.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2916522859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:46.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11"}]: dispatch 2024-09-10T19:04:46.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:46.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11"}]: dispatch 2024-09-10T19:04:46.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:46.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:46.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[40674]: 21.9 deep-scrub starts 2024-09-10T19:04:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[40674]: 21.9 deep-scrub ok 2024-09-10T19:04:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[29577]: pgmap v120: 388 pgs: 32 unknown, 356 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 4 op/s 2024-09-10T19:04:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi119-79823-9"}]': finished 2024-09-10T19:04:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:04:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[29577]: osdmap e105: 8 total, 8 up, 8 in 2024-09-10T19:04:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2916522859' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11"}]: dispatch 2024-09-10T19:04:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11"}]: dispatch 2024-09-10T19:04:46.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:46.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:46.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[29577]: 21.9 deep-scrub starts 2024-09-10T19:04:46.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[29577]: 21.9 deep-scrub ok 2024-09-10T19:04:46.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:46.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:04:47.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:47 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:47.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:47 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:47 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11"}]': finished 2024-09-10T19:04:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:47 smithi194 ceph-mon[32464]: osdmap e106: 8 total, 8 up, 8 in 2024-09-10T19:04:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi119-79823-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1222737451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi119-79477-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:47 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi119-79477-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:04:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:47 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:47.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11"}]': finished 2024-09-10T19:04:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[29577]: osdmap e106: 8 total, 8 up, 8 in 2024-09-10T19:04:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi119-79823-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1222737451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi119-79477-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi119-79477-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:04:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:04:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-11"}]': finished 2024-09-10T19:04:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:04:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[40674]: osdmap e106: 8 total, 8 up, 8 in 2024-09-10T19:04:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi119-79823-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1222737451' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi119-79477-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi119-79477-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:04:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:04:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:47 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:48.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:04:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:04:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:48 smithi194 ceph-mon[32464]: pgmap v123: 388 pgs: 64 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi119-79477-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:48 smithi194 ceph-mon[32464]: osdmap e107: 8 total, 8 up, 8 in 2024-09-10T19:04:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:48 smithi194 ceph-mon[32464]: 21.3 deep-scrub starts 2024-09-10T19:04:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:48 smithi194 ceph-mon[32464]: 21.3 deep-scrub ok 2024-09-10T19:04:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi119-79823-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]': finished 2024-09-10T19:04:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:48 smithi194 ceph-mon[32464]: osdmap e108: 8 total, 8 up, 8 in 2024-09-10T19:04:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1248120655' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi119-79499-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi119-79499-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[29577]: pgmap v123: 388 pgs: 64 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi119-79477-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[29577]: osdmap e107: 8 total, 8 up, 8 in 2024-09-10T19:04:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[29577]: 21.3 deep-scrub starts 2024-09-10T19:04:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[29577]: 21.3 deep-scrub ok 2024-09-10T19:04:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi119-79823-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]': finished 2024-09-10T19:04:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[29577]: osdmap e108: 8 total, 8 up, 8 in 2024-09-10T19:04:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1248120655' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi119-79499-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:48.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi119-79499-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[40674]: pgmap v123: 388 pgs: 64 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi119-79477-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[40674]: osdmap e107: 8 total, 8 up, 8 in 2024-09-10T19:04:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[40674]: 21.3 deep-scrub starts 2024-09-10T19:04:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[40674]: 21.3 deep-scrub ok 2024-09-10T19:04:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi119-79823-14", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]': finished 2024-09-10T19:04:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[40674]: osdmap e108: 8 total, 8 up, 8 in 2024-09-10T19:04:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:48.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1248120655' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi119-79499-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:48.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi119-79499-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:49.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:49 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:49 smithi194 ceph-mon[32464]: 21.1 deep-scrub starts 2024-09-10T19:04:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:49 smithi194 ceph-mon[32464]: 21.1 deep-scrub ok 2024-09-10T19:04:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi119-79499-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:49 smithi194 ceph-mon[32464]: osdmap e109: 8 total, 8 up, 8 in 2024-09-10T19:04:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/107563430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi119-79477-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:49.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:49.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:49.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[29577]: 21.1 deep-scrub starts 2024-09-10T19:04:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[29577]: 21.1 deep-scrub ok 2024-09-10T19:04:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi119-79499-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[29577]: osdmap e109: 8 total, 8 up, 8 in 2024-09-10T19:04:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/107563430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi119-79477-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:49.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:49.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:49.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[40674]: 21.1 deep-scrub starts 2024-09-10T19:04:49.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[40674]: 21.1 deep-scrub ok 2024-09-10T19:04:49.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:49.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi119-79499-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:49.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[40674]: osdmap e109: 8 total, 8 up, 8 in 2024-09-10T19:04:49.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/107563430' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi119-79477-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:49.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:50.461 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:04:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:04:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:04:50.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:50 smithi194 ceph-mon[32464]: pgmap v126: 364 pgs: 72 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:04:50.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:50 smithi194 ceph-mon[32464]: 21.4 deep-scrub starts 2024-09-10T19:04:50.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:50 smithi194 ceph-mon[32464]: 21.4 deep-scrub ok 2024-09-10T19:04:50.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/107563430' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi119-79477-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:50.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:50 smithi194 ceph-mon[32464]: osdmap e110: 8 total, 8 up, 8 in 2024-09-10T19:04:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:50 smithi119 ceph-mon[29577]: pgmap v126: 364 pgs: 72 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:04:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:50 smithi119 ceph-mon[29577]: 21.4 deep-scrub starts 2024-09-10T19:04:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:50 smithi119 ceph-mon[29577]: 21.4 deep-scrub ok 2024-09-10T19:04:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/107563430' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi119-79477-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:50.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:50 smithi119 ceph-mon[29577]: osdmap e110: 8 total, 8 up, 8 in 2024-09-10T19:04:50.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:50 smithi119 ceph-mon[40674]: pgmap v126: 364 pgs: 72 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:04:50.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:50 smithi119 ceph-mon[40674]: 21.4 deep-scrub starts 2024-09-10T19:04:50.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:50 smithi119 ceph-mon[40674]: 21.4 deep-scrub ok 2024-09-10T19:04:50.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/107563430' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi119-79477-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:50.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:50 smithi119 ceph-mon[40674]: osdmap e110: 8 total, 8 up, 8 in 2024-09-10T19:04:51.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:51.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:51 smithi194 ceph-mon[32464]: osdmap e111: 8 total, 8 up, 8 in 2024-09-10T19:04:51.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1802005045' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi119-79499-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:51.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi119-79499-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:51.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:51.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:51 smithi119 ceph-mon[29577]: osdmap e111: 8 total, 8 up, 8 in 2024-09-10T19:04:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1802005045' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi119-79499-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi119-79499-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:51 smithi119 ceph-mon[40674]: osdmap e111: 8 total, 8 up, 8 in 2024-09-10T19:04:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1802005045' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi119-79499-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi119-79499-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:52.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:52 smithi194 ceph-mon[32464]: pgmap v129: 364 pgs: 72 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:04:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:52 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi119-79499-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:52 smithi194 ceph-mon[32464]: osdmap e112: 8 total, 8 up, 8 in 2024-09-10T19:04:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3593900067' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi119-79477-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:52.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:04:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:04:52] "GET /metrics HTTP/1.1" 200 240015 "" "Prometheus/2.43.0" 2024-09-10T19:04:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:52 smithi119 ceph-mon[29577]: pgmap v129: 364 pgs: 72 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:04:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:52 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi119-79499-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:52 smithi119 ceph-mon[29577]: osdmap e112: 8 total, 8 up, 8 in 2024-09-10T19:04:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3593900067' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi119-79477-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:52.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:52 smithi119 ceph-mon[40674]: pgmap v129: 364 pgs: 72 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:04:52.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:52 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:52.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi119-79499-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:52.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:52 smithi119 ceph-mon[40674]: osdmap e112: 8 total, 8 up, 8 in 2024-09-10T19:04:52.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3593900067' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi119-79477-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:52.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:54.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:54 smithi194 ceph-mon[32464]: pgmap v132: 396 pgs: 32 creating+peering, 364 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 104 KiB/s wr, 102 op/s 2024-09-10T19:04:54.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:54 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3593900067' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi119-79477-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:54.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:54 smithi194 ceph-mon[32464]: osdmap e113: 8 total, 8 up, 8 in 2024-09-10T19:04:54.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:54 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:54.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:54 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:54.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:54 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:54.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:54 smithi119 ceph-mon[29577]: pgmap v132: 396 pgs: 32 creating+peering, 364 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 104 KiB/s wr, 102 op/s 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3593900067' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi119-79477-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:54 smithi119 ceph-mon[29577]: osdmap e113: 8 total, 8 up, 8 in 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:54 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:54 smithi119 ceph-mon[40674]: pgmap v132: 396 pgs: 32 creating+peering, 364 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 104 KiB/s wr, 102 op/s 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:54 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3593900067' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi119-79477-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:54 smithi119 ceph-mon[40674]: osdmap e113: 8 total, 8 up, 8 in 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:54 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:54.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:54 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:54.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:54 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:04:55.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:55 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:55.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:55 smithi194 ceph-mon[32464]: osdmap e114: 8 total, 8 up, 8 in 2024-09-10T19:04:55.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-13"}]: dispatch 2024-09-10T19:04:55.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:55 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-13"}]: dispatch 2024-09-10T19:04:55.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2580621363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi119-79499-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:55.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:55.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:55 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-13"}]': finished 2024-09-10T19:04:55.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2580621363' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi119-79499-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:55.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:55 smithi194 ceph-mon[32464]: osdmap e115: 8 total, 8 up, 8 in 2024-09-10T19:04:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[29577]: osdmap e114: 8 total, 8 up, 8 in 2024-09-10T19:04:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-13"}]: dispatch 2024-09-10T19:04:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-13"}]: dispatch 2024-09-10T19:04:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2580621363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi119-79499-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-13"}]': finished 2024-09-10T19:04:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2580621363' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi119-79499-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:55.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[29577]: osdmap e115: 8 total, 8 up, 8 in 2024-09-10T19:04:55.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:04:55.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[40674]: osdmap e114: 8 total, 8 up, 8 in 2024-09-10T19:04:55.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-13"}]: dispatch 2024-09-10T19:04:55.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-13"}]: dispatch 2024-09-10T19:04:55.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2580621363' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi119-79499-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:55.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:55.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-13"}]': finished 2024-09-10T19:04:55.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2580621363' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi119-79499-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:55.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:55 smithi119 ceph-mon[40674]: osdmap e115: 8 total, 8 up, 8 in 2024-09-10T19:04:56.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:56 smithi194 ceph-mon[32464]: pgmap v135: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 102 op/s 2024-09-10T19:04:56.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3696954621' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi119-79477-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:56.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-13", "mode": "writeback"}]: dispatch 2024-09-10T19:04:56.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:56 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi119-79477-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:56.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:56 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-13", "mode": "writeback"}]: dispatch 2024-09-10T19:04:56.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:56.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:56 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:56.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:56 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi119-79477-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:56.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:56 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-13", "mode": "writeback"}]': finished 2024-09-10T19:04:56.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:56 smithi194 ceph-mon[32464]: osdmap e116: 8 total, 8 up, 8 in 2024-09-10T19:04:56.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[29577]: pgmap v135: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 102 op/s 2024-09-10T19:04:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3696954621' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi119-79477-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-13", "mode": "writeback"}]: dispatch 2024-09-10T19:04:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi119-79477-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-13", "mode": "writeback"}]: dispatch 2024-09-10T19:04:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi119-79477-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-13", "mode": "writeback"}]': finished 2024-09-10T19:04:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[29577]: osdmap e116: 8 total, 8 up, 8 in 2024-09-10T19:04:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[40674]: pgmap v135: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 104 KiB/s wr, 102 op/s 2024-09-10T19:04:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3696954621' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi119-79477-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-13", "mode": "writeback"}]: dispatch 2024-09-10T19:04:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi119-79477-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-13", "mode": "writeback"}]: dispatch 2024-09-10T19:04:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:04:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi119-79477-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-13", "mode": "writeback"}]': finished 2024-09-10T19:04:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:56 smithi119 ceph-mon[40674]: osdmap e116: 8 total, 8 up, 8 in 2024-09-10T19:04:57.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-09-10T19:04:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-09-10T19:04:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-09-10T19:04:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-09-10T19:04:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-09-10T19:04:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-09-10T19:04:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-09-10T19:04:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-09-10T19:04:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-09-10T19:04:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-09-10T19:04:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:57.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: osdmap e117: 8 total, 8 up, 8 in 2024-09-10T19:04:57.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2590917387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi119-79499-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:57.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-09-10T19:04:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-09-10T19:04:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-09-10T19:04:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-09-10T19:04:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-09-10T19:04:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-09-10T19:04:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-09-10T19:04:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-09-10T19:04:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-09-10T19:04:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-09-10T19:04:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: osdmap e117: 8 total, 8 up, 8 in 2024-09-10T19:04:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2590917387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi119-79499-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:57.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:57.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-09-10T19:04:57.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-09-10T19:04:57.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-09-10T19:04:57.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-09-10T19:04:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-09-10T19:04:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-09-10T19:04:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-09-10T19:04:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-09-10T19:04:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-09-10T19:04:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-09-10T19:04:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:04:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: osdmap e117: 8 total, 8 up, 8 in 2024-09-10T19:04:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2590917387' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi119-79499-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:58.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:04:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-09-10T19:04:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-09-10T19:04:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-09-10T19:04:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-09-10T19:04:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-09-10T19:04:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-09-10T19:04:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-09-10T19:04:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-09-10T19:04:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-09-10T19:04:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-09-10T19:04:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: 95.1 scrub starts 2024-09-10T19:04:58.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: 95.1 scrub ok 2024-09-10T19:04:58.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: pgmap v138: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:58.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:58.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2590917387' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi119-79499-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:58.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]': finished 2024-09-10T19:04:58.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: osdmap e118: 8 total, 8 up, 8 in 2024-09-10T19:04:58.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/435266286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi119-79477-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:58.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:58.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi119-79477-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:58.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-09-10T19:04:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-09-10T19:04:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-09-10T19:04:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-09-10T19:04:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-09-10T19:04:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-09-10T19:04:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-09-10T19:04:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-09-10T19:04:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-09-10T19:04:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-09-10T19:04:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: 95.1 scrub starts 2024-09-10T19:04:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: 95.1 scrub ok 2024-09-10T19:04:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: pgmap v138: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2590917387' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi119-79499-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]': finished 2024-09-10T19:04:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: osdmap e118: 8 total, 8 up, 8 in 2024-09-10T19:04:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/435266286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi119-79477-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:58.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:58.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi119-79477-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:58.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:58.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.0"}]: dispatch 2024-09-10T19:04:58.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.1"}]: dispatch 2024-09-10T19:04:58.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.2"}]: dispatch 2024-09-10T19:04:58.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.3"}]: dispatch 2024-09-10T19:04:58.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.4"}]: dispatch 2024-09-10T19:04:58.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.5"}]: dispatch 2024-09-10T19:04:58.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.6"}]: dispatch 2024-09-10T19:04:58.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.7"}]: dispatch 2024-09-10T19:04:58.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.8"}]: dispatch 2024-09-10T19:04:58.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "95.9"}]: dispatch 2024-09-10T19:04:58.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: 95.1 scrub starts 2024-09-10T19:04:58.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: 95.1 scrub ok 2024-09-10T19:04:58.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: pgmap v138: 364 pgs: 32 unknown, 332 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:04:58.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:58.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2590917387' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi119-79499-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:58.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]': finished 2024-09-10T19:04:58.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: osdmap e118: 8 total, 8 up, 8 in 2024-09-10T19:04:58.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/435266286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi119-79477-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:58.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]: dispatch 2024-09-10T19:04:58.821 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi119-79477-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:04:58.821 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:59.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-10T19:04:59.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [==========] Running 12 tests from 4 test suites. 2024-09-10T19:04:59.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-10T19:04:59.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-10T19:04:59.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-10T19:04:59.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2245 ms) 2024-09-10T19:04:59.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-10T19:04:59.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (3302 ms) 2024-09-10T19:04:59.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-10T19:04:59.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (1749 ms) 2024-09-10T19:04:59.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-10T19:04:59.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (1676 ms) 2024-09-10T19:04:59.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8972 ms total) 2024-09-10T19:04:59.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: 2024-09-10T19:04:59.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged 2024-09-10T19:04:59.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-10T19:04:59.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4127 ms) 2024-09-10T19:04:59.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-10T19:04:59.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (3994 ms) 2024-09-10T19:04:59.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManaged (8121 ms total) 2024-09-10T19:04:59.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: 2024-09-10T19:04:59.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-10T19:04:59.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-10T19:04:59.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2989 ms) 2024-09-10T19:04:59.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-10T19:04:59.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1898 ms) 2024-09-10T19:04:59.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-10T19:04:59.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1988 ms) 2024-09-10T19:04:59.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-10T19:04:59.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1983 ms) 2024-09-10T19:04:59.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (8858 ms total) 2024-09-10T19:04:59.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: 2024-09-10T19:04:59.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-10T19:04:59.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-10T19:04:59.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4026 ms) 2024-09-10T19:04:59.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-10T19:04:59.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4015 ms) 2024-09-10T19:04:59.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8041 ms total) 2024-09-10T19:04:59.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: 2024-09-10T19:04:59.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-10T19:04:59.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [==========] 12 tests from 4 test suites ran. (57829 ms total) 2024-09-10T19:04:59.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots: [ PASSED ] 12 tests. 2024-09-10T19:04:59.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:59 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:59.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:59 smithi194 ceph-mon[32464]: 95.8 scrub starts 2024-09-10T19:04:59.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:59 smithi194 ceph-mon[32464]: 95.8 scrub ok 2024-09-10T19:04:59.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:59 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]': finished 2024-09-10T19:04:59.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi119-79477-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:59.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:59 smithi194 ceph-mon[32464]: osdmap e119: 8 total, 8 up, 8 in 2024-09-10T19:04:59.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:04:59 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:59.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:59 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:59 smithi119 ceph-mon[29577]: 95.8 scrub starts 2024-09-10T19:04:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:59 smithi119 ceph-mon[29577]: 95.8 scrub ok 2024-09-10T19:04:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:59 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]': finished 2024-09-10T19:04:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi119-79477-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:59 smithi119 ceph-mon[29577]: osdmap e119: 8 total, 8 up, 8 in 2024-09-10T19:04:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:04:59 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:04:59.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:59 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:04:59.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:59 smithi119 ceph-mon[40674]: 95.8 scrub starts 2024-09-10T19:04:59.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:59 smithi119 ceph-mon[40674]: 95.8 scrub ok 2024-09-10T19:04:59.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:59 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3733195072' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi119-79823-14"}]': finished 2024-09-10T19:04:59.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi119-79477-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:04:59.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:59 smithi119 ceph-mon[40674]: osdmap e119: 8 total, 8 up, 8 in 2024-09-10T19:04:59.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:04:59 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:00.457 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:05:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:05:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:05:00.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:00 smithi194 ceph-mon[32464]: pgmap v141: 388 pgs: 32 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 21 KiB/s wr, 100 op/s 2024-09-10T19:05:00.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:00 smithi194 ceph-mon[32464]: 95.3 scrub starts 2024-09-10T19:05:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:00 smithi194 ceph-mon[32464]: 95.3 scrub ok 2024-09-10T19:05:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:00 smithi194 ceph-mon[32464]: osdmap e120: 8 total, 8 up, 8 in 2024-09-10T19:05:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1905779804' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi119-79499-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:00.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:00 smithi119 ceph-mon[29577]: pgmap v141: 388 pgs: 32 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 21 KiB/s wr, 100 op/s 2024-09-10T19:05:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:00 smithi119 ceph-mon[29577]: 95.3 scrub starts 2024-09-10T19:05:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:00 smithi119 ceph-mon[29577]: 95.3 scrub ok 2024-09-10T19:05:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:00 smithi119 ceph-mon[29577]: osdmap e120: 8 total, 8 up, 8 in 2024-09-10T19:05:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1905779804' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi119-79499-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:00 smithi119 ceph-mon[40674]: pgmap v141: 388 pgs: 32 unknown, 32 creating+peering, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 21 KiB/s wr, 100 op/s 2024-09-10T19:05:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:00 smithi119 ceph-mon[40674]: 95.3 scrub starts 2024-09-10T19:05:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:00 smithi119 ceph-mon[40674]: 95.3 scrub ok 2024-09-10T19:05:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:00 smithi119 ceph-mon[40674]: osdmap e120: 8 total, 8 up, 8 in 2024-09-10T19:05:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1905779804' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi119-79499-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:01.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:01 smithi194 ceph-mon[32464]: 95.6 scrub starts 2024-09-10T19:05:01.772 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:01 smithi194 ceph-mon[32464]: 95.6 scrub ok 2024-09-10T19:05:01.772 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1905779804' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi119-79499-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:01.772 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:01 smithi194 ceph-mon[32464]: osdmap e121: 8 total, 8 up, 8 in 2024-09-10T19:05:01.772 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2617927687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi119-79477-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:01.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:01 smithi119 ceph-mon[29577]: 95.6 scrub starts 2024-09-10T19:05:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:01 smithi119 ceph-mon[29577]: 95.6 scrub ok 2024-09-10T19:05:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1905779804' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi119-79499-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:01 smithi119 ceph-mon[29577]: osdmap e121: 8 total, 8 up, 8 in 2024-09-10T19:05:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2617927687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi119-79477-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:01 smithi119 ceph-mon[40674]: 95.6 scrub starts 2024-09-10T19:05:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:01 smithi119 ceph-mon[40674]: 95.6 scrub ok 2024-09-10T19:05:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1905779804' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi119-79499-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:01 smithi119 ceph-mon[40674]: osdmap e121: 8 total, 8 up, 8 in 2024-09-10T19:05:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2617927687' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi119-79477-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:02.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:02 smithi194 ceph-mon[32464]: pgmap v144: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 21 KiB/s wr, 100 op/s 2024-09-10T19:05:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:02 smithi194 ceph-mon[32464]: 95.5 scrub starts 2024-09-10T19:05:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:02 smithi194 ceph-mon[32464]: 95.5 scrub ok 2024-09-10T19:05:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:02 smithi194 ceph-mon[32464]: 95.7 scrub starts 2024-09-10T19:05:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:02 smithi194 ceph-mon[32464]: 95.7 scrub ok 2024-09-10T19:05:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:05:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:02 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2617927687' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi119-79477-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:02 smithi194 ceph-mon[32464]: osdmap e122: 8 total, 8 up, 8 in 2024-09-10T19:05:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1117783486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79837-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:02 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79837-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:02.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:02.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:05:02] "GET /metrics HTTP/1.1" 200 237860 "" "Prometheus/2.43.0" 2024-09-10T19:05:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[29577]: pgmap v144: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 21 KiB/s wr, 100 op/s 2024-09-10T19:05:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[29577]: 95.5 scrub starts 2024-09-10T19:05:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[29577]: 95.5 scrub ok 2024-09-10T19:05:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[29577]: 95.7 scrub starts 2024-09-10T19:05:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[29577]: 95.7 scrub ok 2024-09-10T19:05:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:05:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2617927687' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi119-79477-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[29577]: osdmap e122: 8 total, 8 up, 8 in 2024-09-10T19:05:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1117783486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79837-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79837-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:02.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:02.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[40674]: pgmap v144: 356 pgs: 32 unknown, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 55 KiB/s rd, 21 KiB/s wr, 100 op/s 2024-09-10T19:05:02.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:02.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[40674]: 95.5 scrub starts 2024-09-10T19:05:02.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[40674]: 95.5 scrub ok 2024-09-10T19:05:02.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[40674]: 95.7 scrub starts 2024-09-10T19:05:02.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[40674]: 95.7 scrub ok 2024-09-10T19:05:02.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:05:02.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:02.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:02.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2617927687' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi119-79477-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:02.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[40674]: osdmap e122: 8 total, 8 up, 8 in 2024-09-10T19:05:02.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1117783486' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79837-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:02.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79837-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:02.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:04.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:04 smithi194 ceph-mon[32464]: pgmap v147: 356 pgs: 32 creating+peering, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:05:04.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:04 smithi194 ceph-mon[32464]: 95.2 scrub starts 2024-09-10T19:05:04.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:04 smithi194 ceph-mon[32464]: 95.2 scrub ok 2024-09-10T19:05:04.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79837-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:04.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:04 smithi194 ceph-mon[32464]: osdmap e123: 8 total, 8 up, 8 in 2024-09-10T19:05:04.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4060016240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi119-79499-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:04.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:04 smithi119 ceph-mon[29577]: pgmap v147: 356 pgs: 32 creating+peering, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:05:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:04 smithi119 ceph-mon[29577]: 95.2 scrub starts 2024-09-10T19:05:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:04 smithi119 ceph-mon[29577]: 95.2 scrub ok 2024-09-10T19:05:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79837-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:04 smithi119 ceph-mon[29577]: osdmap e123: 8 total, 8 up, 8 in 2024-09-10T19:05:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4060016240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi119-79499-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:04 smithi119 ceph-mon[40674]: pgmap v147: 356 pgs: 32 creating+peering, 324 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:05:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:04 smithi119 ceph-mon[40674]: 95.2 scrub starts 2024-09-10T19:05:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:04 smithi119 ceph-mon[40674]: 95.2 scrub ok 2024-09-10T19:05:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79837-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:04 smithi119 ceph-mon[40674]: osdmap e123: 8 total, 8 up, 8 in 2024-09-10T19:05:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4060016240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi119-79499-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:05.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4060016240' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi119-79499-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:05.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:05 smithi194 ceph-mon[32464]: osdmap e124: 8 total, 8 up, 8 in 2024-09-10T19:05:05.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3327474879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79477-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79477-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:05 smithi194 ceph-mon[32464]: 95.0 scrub starts 2024-09-10T19:05:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:05 smithi194 ceph-mon[32464]: 95.0 scrub ok 2024-09-10T19:05:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79477-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:05 smithi194 ceph-mon[32464]: osdmap e125: 8 total, 8 up, 8 in 2024-09-10T19:05:05.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4060016240' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi119-79499-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[29577]: osdmap e124: 8 total, 8 up, 8 in 2024-09-10T19:05:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3327474879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79477-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79477-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[29577]: 95.0 scrub starts 2024-09-10T19:05:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[29577]: 95.0 scrub ok 2024-09-10T19:05:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79477-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[29577]: osdmap e125: 8 total, 8 up, 8 in 2024-09-10T19:05:05.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4060016240' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi119-79499-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[40674]: osdmap e124: 8 total, 8 up, 8 in 2024-09-10T19:05:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3327474879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79477-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79477-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[40674]: 95.0 scrub starts 2024-09-10T19:05:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[40674]: 95.0 scrub ok 2024-09-10T19:05:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79477-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:05 smithi119 ceph-mon[40674]: osdmap e125: 8 total, 8 up, 8 in 2024-09-10T19:05:06.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:06 smithi194 ceph-mon[32464]: pgmap v150: 388 pgs: 64 unknown, 32 creating+peering, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:05:06.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:06.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:06 smithi194 ceph-mon[32464]: osdmap e126: 8 total, 8 up, 8 in 2024-09-10T19:05:06.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:06 smithi119 ceph-mon[29577]: pgmap v150: 388 pgs: 64 unknown, 32 creating+peering, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:05:06.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:06.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:06 smithi119 ceph-mon[29577]: osdmap e126: 8 total, 8 up, 8 in 2024-09-10T19:05:06.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:06 smithi119 ceph-mon[40674]: pgmap v150: 388 pgs: 64 unknown, 32 creating+peering, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:05:06.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:06.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:06 smithi119 ceph-mon[40674]: osdmap e126: 8 total, 8 up, 8 in 2024-09-10T19:05:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1755480163' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi119-79499-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi119-79499-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:07.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:07.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi119-79499-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:07.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:07.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: osdmap e127: 8 total, 8 up, 8 in 2024-09-10T19:05:07.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi119-79837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/246258029' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi119-79477-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:07.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi119-79837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi119-79477-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1755480163' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi119-79499-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi119-79499-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:07.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi119-79499-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: osdmap e127: 8 total, 8 up, 8 in 2024-09-10T19:05:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi119-79837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/246258029' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi119-79477-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:07.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi119-79837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi119-79477-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1755480163' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi119-79499-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi119-79499-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:07.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:07.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi119-79499-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:07.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:07.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: osdmap e127: 8 total, 8 up, 8 in 2024-09-10T19:05:07.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi119-79837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/246258029' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi119-79477-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:07.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi119-79837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:07.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi119-79477-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:08.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:05:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:05:08.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:08 smithi194 ceph-mon[32464]: pgmap v153: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:05:08.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:08.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:08 smithi119 ceph-mon[29577]: pgmap v153: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:05:08.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:08 smithi119 ceph-mon[40674]: pgmap v153: 324 pgs: 32 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:05:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:09.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:09 smithi194 ceph-mon[32464]: 95.9 scrub starts 2024-09-10T19:05:09.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:09 smithi194 ceph-mon[32464]: 95.9 scrub ok 2024-09-10T19:05:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:09 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi119-79477-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:09 smithi194 ceph-mon[32464]: osdmap e128: 8 total, 8 up, 8 in 2024-09-10T19:05:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi119-79837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]': finished 2024-09-10T19:05:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:09 smithi194 ceph-mon[32464]: osdmap e129: 8 total, 8 up, 8 in 2024-09-10T19:05:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2221138544' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[29577]: 95.9 scrub starts 2024-09-10T19:05:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[29577]: 95.9 scrub ok 2024-09-10T19:05:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi119-79477-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[29577]: osdmap e128: 8 total, 8 up, 8 in 2024-09-10T19:05:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi119-79837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]': finished 2024-09-10T19:05:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[29577]: osdmap e129: 8 total, 8 up, 8 in 2024-09-10T19:05:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2221138544' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[40674]: 95.9 scrub starts 2024-09-10T19:05:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[40674]: 95.9 scrub ok 2024-09-10T19:05:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi119-79477-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[40674]: osdmap e128: 8 total, 8 up, 8 in 2024-09-10T19:05:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi119-79837-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]': finished 2024-09-10T19:05:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[40674]: osdmap e129: 8 total, 8 up, 8 in 2024-09-10T19:05:09.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2221138544' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:09.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:10.489 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:05:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:05:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:05:10.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:10 smithi194 ceph-mon[32464]: pgmap v156: 324 pgs: 32 creating+peering, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:10 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:10 smithi194 ceph-mon[32464]: osdmap e130: 8 total, 8 up, 8 in 2024-09-10T19:05:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1739740625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi119-79477-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi119-79477-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[29577]: pgmap v156: 324 pgs: 32 creating+peering, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[29577]: osdmap e130: 8 total, 8 up, 8 in 2024-09-10T19:05:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1739740625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi119-79477-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi119-79477-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[40674]: pgmap v156: 324 pgs: 32 creating+peering, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[40674]: osdmap e130: 8 total, 8 up, 8 in 2024-09-10T19:05:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1739740625' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi119-79477-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi119-79477-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:11.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T19:05:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi119-79477-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:11 smithi194 ceph-mon[32464]: osdmap e131: 8 total, 8 up, 8 in 2024-09-10T19:05:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app2"}]: dispatch 2024-09-10T19:05:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:11.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T19:05:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi119-79477-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[29577]: osdmap e131: 8 total, 8 up, 8 in 2024-09-10T19:05:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app2"}]: dispatch 2024-09-10T19:05:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T19:05:11.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:11.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:11.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi119-79477-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:11.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:11.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[40674]: osdmap e131: 8 total, 8 up, 8 in 2024-09-10T19:05:11.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app2"}]: dispatch 2024-09-10T19:05:11.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:11.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:12.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:12 smithi194 ceph-mon[32464]: 95.4 scrub starts 2024-09-10T19:05:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:12 smithi194 ceph-mon[32464]: 95.4 scrub ok 2024-09-10T19:05:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:12 smithi194 ceph-mon[32464]: pgmap v159: 364 pgs: 72 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:12 smithi194 ceph-mon[32464]: osdmap e132: 8 total, 8 up, 8 in 2024-09-10T19:05:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1886093946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi119-79499-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:05:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:05:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:05:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:12.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:05:12] "GET /metrics HTTP/1.1" 200 237860 "" "Prometheus/2.43.0" 2024-09-10T19:05:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[29577]: 95.4 scrub starts 2024-09-10T19:05:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[29577]: 95.4 scrub ok 2024-09-10T19:05:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[29577]: pgmap v159: 364 pgs: 72 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[29577]: osdmap e132: 8 total, 8 up, 8 in 2024-09-10T19:05:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1886093946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi119-79499-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:12.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:05:12.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:05:12.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:05:12.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:12.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[40674]: 95.4 scrub starts 2024-09-10T19:05:12.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[40674]: 95.4 scrub ok 2024-09-10T19:05:12.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[40674]: pgmap v159: 364 pgs: 72 unknown, 292 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:12.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:12.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi119-79801-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:12.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[40674]: osdmap e132: 8 total, 8 up, 8 in 2024-09-10T19:05:12.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1886093946' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi119-79499-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:12.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:05:12.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:05:12.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-09-10T19:05:12.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:14.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:14 smithi194 ceph-mon[32464]: pgmap v162: 332 pgs: 32 creating+peering, 300 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:05:14.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1886093946' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi119-79499-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:14.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-10T19:05:14.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:14 smithi194 ceph-mon[32464]: osdmap e133: 8 total, 8 up, 8 in 2024-09-10T19:05:14.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/565587736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi119-79477-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:14.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi119-79477-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:14.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T19:05:14.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T19:05:14.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[29577]: pgmap v162: 332 pgs: 32 creating+peering, 300 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:05:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1886093946' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi119-79499-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-10T19:05:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[29577]: osdmap e133: 8 total, 8 up, 8 in 2024-09-10T19:05:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/565587736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi119-79477-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi119-79477-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T19:05:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T19:05:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:14.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[40674]: pgmap v162: 332 pgs: 32 creating+peering, 300 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:05:14.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1886093946' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi119-79499-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:14.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-09-10T19:05:14.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[40674]: osdmap e133: 8 total, 8 up, 8 in 2024-09-10T19:05:14.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/565587736' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi119-79477-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:14.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi119-79477-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:14.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T19:05:14.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-09-10T19:05:14.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:15.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi119-79477-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:15.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-10T19:05:15.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:15 smithi194 ceph-mon[32464]: osdmap e134: 8 total, 8 up, 8 in 2024-09-10T19:05:15.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T19:05:15.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T19:05:15.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:15.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi119-79477-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-10T19:05:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:15 smithi119 ceph-mon[29577]: osdmap e134: 8 total, 8 up, 8 in 2024-09-10T19:05:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T19:05:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T19:05:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi119-79477-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-09-10T19:05:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:15 smithi119 ceph-mon[40674]: osdmap e134: 8 total, 8 up, 8 in 2024-09-10T19:05:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3897258915' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T19:05:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1"}]: dispatch 2024-09-10T19:05:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:16.250 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-10T19:05:16.251 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-10T19:05:16.251 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-10T19:05:16.251 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-10T19:05:16.251 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-10T19:05:16.251 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-10T19:05:16.251 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: 2024-09-10T19:05:16.251 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-10T19:05:16.251 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: seed 79801 2024-09-10T19:05:16.251 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-10T19:05:16.251 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-09-10T19:05:16.251 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-10T19:05:16.251 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (233 ms) 2024-09-10T19:05:16.252 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-10T19:05:16.252 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (12 ms) 2024-09-10T19:05:16.252 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-10T19:05:16.252 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (6 ms) 2024-09-10T19:05:16.252 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-10T19:05:16.252 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (9 ms) 2024-09-10T19:05:16.252 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-10T19:05:16.252 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (2 ms) 2024-09-10T19:05:16.252 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-10T19:05:16.252 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (1 ms) 2024-09-10T19:05:16.252 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-10T19:05:16.252 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (4 ms) 2024-09-10T19:05:16.252 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (7 ms) 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (3 ms) 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (6 ms) 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1389 ms) 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-10T19:05:16.253 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (389 ms) 2024-09-10T19:05:16.254 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-10T19:05:16.254 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-10T19:05:16.254 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: done waiting, doing copies 2024-09-10T19:05:16.254 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-10T19:05:16.254 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: done waiting 2024-09-10T19:05:16.254 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60725 ms) 2024-09-10T19:05:16.254 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-10T19:05:16.254 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (5 ms) 2024-09-10T19:05:16.254 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-10T19:05:16.254 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (3 ms) 2024-09-10T19:05:16.254 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-10T19:05:16.254 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4312 ms) 2024-09-10T19:05:16.254 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-10T19:05:16.255 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-10T19:05:16.255 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-10T19:05:16.255 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-09-10T19:05:16.255 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-10T19:05:16.255 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-10T19:05:16.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:16 smithi194 ceph-mon[32464]: pgmap v165: 332 pgs: 32 unknown, 300 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:05:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1"}]': finished 2024-09-10T19:05:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:16 smithi194 ceph-mon[32464]: osdmap e135: 8 total, 8 up, 8 in 2024-09-10T19:05:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:16 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/742913650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi119-79499-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/742913650' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi119-79499-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:16 smithi194 ceph-mon[32464]: osdmap e136: 8 total, 8 up, 8 in 2024-09-10T19:05:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2283109537' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi119-79477-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:16.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi119-79477-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[29577]: pgmap v165: 332 pgs: 32 unknown, 300 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:05:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1"}]': finished 2024-09-10T19:05:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[29577]: osdmap e135: 8 total, 8 up, 8 in 2024-09-10T19:05:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/742913650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi119-79499-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/742913650' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi119-79499-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[29577]: osdmap e136: 8 total, 8 up, 8 in 2024-09-10T19:05:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2283109537' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi119-79477-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:16.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi119-79477-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:16.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[40674]: pgmap v165: 332 pgs: 32 unknown, 300 active+clean; 217 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:05:16.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi119-79801-1","app":"app1","key":"key1"}]': finished 2024-09-10T19:05:16.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[40674]: osdmap e135: 8 total, 8 up, 8 in 2024-09-10T19:05:16.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:16.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/742913650' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi119-79499-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:16.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:16.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/742913650' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi119-79499-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:16.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[40674]: osdmap e136: 8 total, 8 up, 8 in 2024-09-10T19:05:16.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2283109537' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi119-79477-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:16.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi119-79477-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79801-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi119-79477-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79801-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:17 smithi194 ceph-mon[32464]: osdmap e137: 8 total, 8 up, 8 in 2024-09-10T19:05:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79801-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:17.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79801-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi119-79477-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79801-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[29577]: osdmap e137: 8 total, 8 up, 8 in 2024-09-10T19:05:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79801-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:17.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79801-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi119-79477-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79801-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[40674]: osdmap e137: 8 total, 8 up, 8 in 2024-09-10T19:05:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79801-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:18.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:05:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:05:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:18 smithi119 ceph-mon[29577]: pgmap v168: 332 pgs: 64 unknown, 268 active+clean; 592 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:05:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:18 smithi119 ceph-mon[29577]: osdmap e138: 8 total, 8 up, 8 in 2024-09-10T19:05:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/208461634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi119-79499-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi119-79499-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:18 smithi119 ceph-mon[40674]: pgmap v168: 332 pgs: 64 unknown, 268 active+clean; 592 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:05:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:18.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:18.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:18 smithi119 ceph-mon[40674]: osdmap e138: 8 total, 8 up, 8 in 2024-09-10T19:05:18.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/208461634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi119-79499-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:18.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi119-79499-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:19.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:18 smithi194 ceph-mon[32464]: pgmap v168: 332 pgs: 64 unknown, 268 active+clean; 592 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-10T19:05:19.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:19.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:19.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:18 smithi194 ceph-mon[32464]: osdmap e138: 8 total, 8 up, 8 in 2024-09-10T19:05:19.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/208461634' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi119-79499-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:19.021 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi119-79499-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:20.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:19 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:20.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:19 smithi194 ceph-mon[32464]: pgmap v171: 300 pgs: 32 unknown, 268 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79801-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79801-24"}]': finished 2024-09-10T19:05:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi119-79499-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:19 smithi194 ceph-mon[32464]: osdmap e139: 8 total, 8 up, 8 in 2024-09-10T19:05:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2589152250' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi119-79477-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi119-79477-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:20.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[29577]: pgmap v171: 300 pgs: 32 unknown, 268 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79801-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79801-24"}]': finished 2024-09-10T19:05:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi119-79499-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[29577]: osdmap e139: 8 total, 8 up, 8 in 2024-09-10T19:05:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2589152250' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi119-79477-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi119-79477-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[40674]: pgmap v171: 300 pgs: 32 unknown, 268 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79801-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79801-24"}]': finished 2024-09-10T19:05:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi119-79499-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[40674]: osdmap e139: 8 total, 8 up, 8 in 2024-09-10T19:05:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2589152250' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi119-79477-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:20.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:20.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi119-79477-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:20.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:20.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:05:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:05:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:05:21.194 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP (6 api_snapshots_pp: Running main() from gmock_main.cc 2024-09-10T19:05:21.194 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [==========] Running 20 tests from 5 test suites. 2024-09-10T19:05:21.194 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-10T19:05:21.194 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-10T19:05:21.194 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: seed 79837 2024-09-10T19:05:21.194 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-10T19:05:21.194 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2496 ms) 2024-09-10T19:05:21.194 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-10T19:05:21.194 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2020 ms) 2024-09-10T19:05:21.194 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-10T19:05:21.194 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2021 ms) 2024-09-10T19:05:21.194 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-10T19:05:21.195 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (1747 ms) 2024-09-10T19:05:21.195 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-10T19:05:21.195 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3680 ms) 2024-09-10T19:05:21.195 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11964 ms total) 2024-09-10T19:05:21.195 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: 2024-09-10T19:05:21.195 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP 2024-09-10T19:05:21.195 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-10T19:05:21.195 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4051 ms) 2024-09-10T19:05:21.195 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-10T19:05:21.195 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4164 ms) 2024-09-10T19:05:21.195 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-10T19:05:21.195 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (5911 ms) 2024-09-10T19:05:21.195 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-10T19:05:21.195 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3875 ms) 2024-09-10T19:05:21.196 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-10T19:05:21.196 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2012 ms) 2024-09-10T19:05:21.196 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-10T19:05:21.196 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi119-79837-7 2024-09-10T19:05:21.196 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-10T19:05:21.196 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18011 ms) 2024-09-10T19:05:21.196 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [----------] 6 tests from LibRadosSnapshotsSelfManagedPP (38024 ms total) 2024-09-10T19:05:21.196 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: 2024-09-10T19:05:21.196 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-10T19:05:21.196 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-10T19:05:21.196 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (9 ms) 2024-09-10T19:05:21.196 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-10T19:05:21.196 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5063 ms) 2024-09-10T19:05:21.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5072 ms total) 2024-09-10T19:05:21.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: 2024-09-10T19:05:21.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-10T19:05:21.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-10T19:05:21.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2989 ms) 2024-09-10T19:05:21.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-10T19:05:21.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2004 ms) 2024-09-10T19:05:21.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-10T19:05:21.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2006 ms) 2024-09-10T19:05:21.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-10T19:05:21.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2185 ms) 2024-09-10T19:05:21.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9184 ms total) 2024-09-10T19:05:21.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: 2024-09-10T19:05:21.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:21 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi119-79477-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:21.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:21 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]': finished 2024-09-10T19:05:21.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:21 smithi194 ceph-mon[32464]: osdmap e140: 8 total, 8 up, 8 in 2024-09-10T19:05:21.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79801-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:21.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:21.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:21 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:21.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:21.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:21 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi119-79477-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:21.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:21 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]': finished 2024-09-10T19:05:21.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:21 smithi119 ceph-mon[29577]: osdmap e140: 8 total, 8 up, 8 in 2024-09-10T19:05:21.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79801-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:21.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:21.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:21 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:21.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:21.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:21 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi119-79477-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:21.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:21 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi119-79837-15"}]': finished 2024-09-10T19:05:21.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:21 smithi119 ceph-mon[40674]: osdmap e140: 8 total, 8 up, 8 in 2024-09-10T19:05:21.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79801-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:21.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/357037669' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:21.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:21 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]: dispatch 2024-09-10T19:05:21.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:22.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: pgmap v174: 332 pgs: 72 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79801-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]': finished 2024-09-10T19:05:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: osdmap e141: 8 total, 8 up, 8 in 2024-09-10T19:05:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79801-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/694379834' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi119-79499-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi119-79477-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:22.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:22.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:22.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:22.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:22.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:22 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: pgmap v174: 332 pgs: 72 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79801-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]': finished 2024-09-10T19:05:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: osdmap e141: 8 total, 8 up, 8 in 2024-09-10T19:05:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79801-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/694379834' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi119-79499-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi119-79477-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:22.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: pgmap v174: 332 pgs: 72 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:22.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79801-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:22.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi119-79837-15"}]': finished 2024-09-10T19:05:22.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: osdmap e141: 8 total, 8 up, 8 in 2024-09-10T19:05:22.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79801-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:22.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/694379834' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi119-79499-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi119-79477-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:23.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:05:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:05:22] "GET /metrics HTTP/1.1" 200 237970 "" "Prometheus/2.43.0" 2024-09-10T19:05:24.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79801-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/694379834' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi119-79499-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi119-79477-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:23 smithi194 ceph-mon[32464]: osdmap e142: 8 total, 8 up, 8 in 2024-09-10T19:05:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi119-79477-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:23 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-09-10T19:05:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79801-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/694379834' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi119-79499-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi119-79477-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[29577]: osdmap e142: 8 total, 8 up, 8 in 2024-09-10T19:05:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi119-79477-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-09-10T19:05:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79801-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/694379834' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi119-79499-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi119-79477-26", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[40674]: osdmap e142: 8 total, 8 up, 8 in 2024-09-10T19:05:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi119-79477-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:23 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-09-10T19:05:25.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:24 smithi194 ceph-mon[32464]: pgmap v177: 332 pgs: 332 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 2.0 MiB/s wr, 1 op/s 2024-09-10T19:05:25.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:25.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:24 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-10T19:05:25.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:24 smithi194 ceph-mon[32464]: osdmap e143: 8 total, 8 up, 8 in 2024-09-10T19:05:25.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi119-79477-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi119-79477-26"}]': finished 2024-09-10T19:05:25.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]': finished 2024-09-10T19:05:25.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:24 smithi194 ceph-mon[32464]: osdmap e144: 8 total, 8 up, 8 in 2024-09-10T19:05:25.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3397832373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79499-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:25.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79499-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:25.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:25.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:25.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:25.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T19:05:25.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T19:05:25.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[29577]: pgmap v177: 332 pgs: 332 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 2.0 MiB/s wr, 1 op/s 2024-09-10T19:05:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-10T19:05:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[29577]: osdmap e143: 8 total, 8 up, 8 in 2024-09-10T19:05:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi119-79477-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi119-79477-26"}]': finished 2024-09-10T19:05:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]': finished 2024-09-10T19:05:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[29577]: osdmap e144: 8 total, 8 up, 8 in 2024-09-10T19:05:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3397832373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79499-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79499-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T19:05:25.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T19:05:25.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[40674]: pgmap v177: 332 pgs: 332 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 2.0 MiB/s wr, 1 op/s 2024-09-10T19:05:25.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:25.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-09-10T19:05:25.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[40674]: osdmap e143: 8 total, 8 up, 8 in 2024-09-10T19:05:25.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi119-79477-26", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi119-79477-26"}]': finished 2024-09-10T19:05:25.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]': finished 2024-09-10T19:05:25.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[40674]: osdmap e144: 8 total, 8 up, 8 in 2024-09-10T19:05:25.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3397832373' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79499-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:25.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79499-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:25.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:25.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:25.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:25.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T19:05:25.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]: dispatch 2024-09-10T19:05:26.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:26 smithi194 ceph-mon[32464]: pgmap v180: 308 pgs: 48 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79499-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79801-24"}]': finished 2024-09-10T19:05:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]': finished 2024-09-10T19:05:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:26 smithi194 ceph-mon[32464]: osdmap e145: 8 total, 8 up, 8 in 2024-09-10T19:05:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T19:05:26.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T19:05:26.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[29577]: pgmap v180: 308 pgs: 48 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:26.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79499-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:26.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79801-24"}]': finished 2024-09-10T19:05:26.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]': finished 2024-09-10T19:05:26.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[29577]: osdmap e145: 8 total, 8 up, 8 in 2024-09-10T19:05:26.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:26.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:26.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:26.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T19:05:26.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T19:05:26.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[40674]: pgmap v180: 308 pgs: 48 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:26.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi119-79499-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:26.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79801-24"}]': finished 2024-09-10T19:05:26.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pg_num","val":"11"}]': finished 2024-09-10T19:05:26.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[40674]: osdmap e145: 8 total, 8 up, 8 in 2024-09-10T19:05:26.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:26.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:26.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79801-24"}]: dispatch 2024-09-10T19:05:26.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T19:05:26.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]: dispatch 2024-09-10T19:05:27.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79801-24"}]': finished 2024-09-10T19:05:27.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:27 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]': finished 2024-09-10T19:05:27.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:27 smithi194 ceph-mon[32464]: osdmap e146: 8 total, 8 up, 8 in 2024-09-10T19:05:27.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:27.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:27.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:27 smithi194 ceph-mon[32464]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:27.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79801-24"}]': finished 2024-09-10T19:05:27.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:27 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]': finished 2024-09-10T19:05:27.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:27 smithi119 ceph-mon[29577]: osdmap e146: 8 total, 8 up, 8 in 2024-09-10T19:05:27.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:27.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:27.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:27 smithi119 ceph-mon[29577]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:27.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1137619112' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79801-24"}]': finished 2024-09-10T19:05:27.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:27 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi119-79624-1","var":"pgp_num","val":"11"}]': finished 2024-09-10T19:05:27.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:27 smithi119 ceph-mon[40674]: osdmap e146: 8 total, 8 up, 8 in 2024-09-10T19:05:27.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:27.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:27.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:27 smithi119 ceph-mon[40674]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:28.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp api_aio: Running main() from gmock_main.cc 2024-09-10T19:05:28.212 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [==========] Running 41 tests from 2 test suites. 2024-09-10T19:05:28.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [----------] Global test environment set-up. 2024-09-10T19:05:28.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [----------] 25 tests from LibRadosAio 2024-09-10T19:05:28.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-10T19:05:28.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.TooBig (6875 ms) 2024-09-10T19:05:28.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-10T19:05:28.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (4784 ms) 2024-09-10T19:05:28.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-10T19:05:28.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (2519 ms) 2024-09-10T19:05:28.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-10T19:05:28.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (2683 ms) 2024-09-10T19:05:28.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-10T19:05:28.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3334 ms) 2024-09-10T19:05:28.213 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-10T19:05:28.214 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (2755 ms) 2024-09-10T19:05:28.214 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-10T19:05:28.214 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3015 ms) 2024-09-10T19:05:28.214 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-10T19:05:28.214 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3195 ms) 2024-09-10T19:05:28.214 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-10T19:05:28.214 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (2895 ms) 2024-09-10T19:05:28.214 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-10T19:05:28.214 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3013 ms) 2024-09-10T19:05:28.214 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-10T19:05:28.214 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2859 ms) 2024-09-10T19:05:28.214 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-10T19:05:28.214 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2993 ms) 2024-09-10T19:05:28.214 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-10T19:05:28.215 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3006 ms) 2024-09-10T19:05:28.215 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-10T19:05:28.215 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3002 ms) 2024-09-10T19:05:28.215 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-10T19:05:28.215 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.Flush (3003 ms) 2024-09-10T19:05:28.215 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-10T19:05:28.215 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2999 ms) 2024-09-10T19:05:28.215 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-10T19:05:28.215 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2992 ms) 2024-09-10T19:05:28.215 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-10T19:05:28.215 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3012 ms) 2024-09-10T19:05:28.215 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-10T19:05:28.215 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3008 ms) 2024-09-10T19:05:28.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-10T19:05:28.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3024 ms) 2024-09-10T19:05:28.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-10T19:05:28.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3008 ms) 2024-09-10T19:05:28.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-10T19:05:28.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3006 ms) 2024-09-10T19:05:28.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-10T19:05:28.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (3009 ms) 2024-09-10T19:05:28.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-10T19:05:28.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3184 ms) 2024-09-10T19:05:28.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-10T19:05:28.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (2771 ms) 2024-09-10T19:05:28.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [----------] 25 tests from LibRadosAio (79945 ms total) 2024-09-10T19:05:28.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: 2024-09-10T19:05:28.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-10T19:05:28.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-09-10T19:05:28.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7013 ms) 2024-09-10T19:05:28.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:05:28.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-mon[32464]: pgmap v183: 268 pgs: 8 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi119-79477-26"}]': finished 2024-09-10T19:05:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-mon[32464]: osdmap e147: 8 total, 8 up, 8 in 2024-09-10T19:05:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/350729663' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi119-79499-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3399941775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi119-79801-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi119-79499-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi119-79801-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi119-79477-26"}]': finished 2024-09-10T19:05:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi119-79499-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi119-79801-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-mon[32464]: osdmap e148: 8 total, 8 up, 8 in 2024-09-10T19:05:28.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[29577]: pgmap v183: 268 pgs: 8 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi119-79477-26"}]': finished 2024-09-10T19:05:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[29577]: osdmap e147: 8 total, 8 up, 8 in 2024-09-10T19:05:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/350729663' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi119-79499-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3399941775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi119-79801-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi119-79499-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi119-79801-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi119-79477-26"}]': finished 2024-09-10T19:05:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi119-79499-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi119-79801-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[29577]: osdmap e148: 8 total, 8 up, 8 in 2024-09-10T19:05:28.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:28.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[40674]: pgmap v183: 268 pgs: 8 unknown, 260 active+clean; 592 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:28.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi119-79477-26"}]': finished 2024-09-10T19:05:28.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:28.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[40674]: osdmap e147: 8 total, 8 up, 8 in 2024-09-10T19:05:28.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/350729663' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi119-79499-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:28.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3399941775' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi119-79801-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:28.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi119-79477-26"}]: dispatch 2024-09-10T19:05:28.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi119-79499-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:28.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi119-79801-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:28.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3491355834' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi119-79477-26"}]': finished 2024-09-10T19:05:28.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi119-79499-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:28.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi119-79801-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:28.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[40674]: osdmap e148: 8 total, 8 up, 8 in 2024-09-10T19:05:28.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:29.237 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:29 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi119-79477-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi119-79477-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:29 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T19:05:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi119-79477-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:29.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:29 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-10T19:05:29.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:29 smithi194 ceph-mon[32464]: osdmap e149: 8 total, 8 up, 8 in 2024-09-10T19:05:29.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:29.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi119-79477-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi119-79477-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi119-79477-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi119-79477-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T19:05:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi119-79477-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-10T19:05:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[29577]: osdmap e149: 8 total, 8 up, 8 in 2024-09-10T19:05:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi119-79477-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi119-79477-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:29.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi119-79477-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:29.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi119-79477-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:29.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-09-10T19:05:29.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi119-79477-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-09-10T19:05:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[40674]: osdmap e149: 8 total, 8 up, 8 in 2024-09-10T19:05:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi119-79477-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi119-79477-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:30.459 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:05:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:05:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:05:30.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:30 smithi194 ceph-mon[32464]: pgmap v186: 332 pgs: 64 creating+peering, 268 active+clean; 606 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 8.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T19:05:30.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:30 smithi119 ceph-mon[29577]: pgmap v186: 332 pgs: 64 creating+peering, 268 active+clean; 606 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 8.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T19:05:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:30 smithi119 ceph-mon[40674]: pgmap v186: 332 pgs: 64 creating+peering, 268 active+clean; 606 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 8.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T19:05:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:31 smithi194 ceph-mon[32464]: osdmap e150: 8 total, 8 up, 8 in 2024-09-10T19:05:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/714028524' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi119-79801-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3566911626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi119-79499-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi119-79477-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi119-79477-27"}]': finished 2024-09-10T19:05:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/714028524' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi119-79801-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3566911626' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi119-79499-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:05:31.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:31 smithi194 ceph-mon[32464]: osdmap e151: 8 total, 8 up, 8 in 2024-09-10T19:05:31.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13"}]: dispatch 2024-09-10T19:05:31.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13"}]: dispatch 2024-09-10T19:05:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[40674]: osdmap e150: 8 total, 8 up, 8 in 2024-09-10T19:05:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/714028524' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi119-79801-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3566911626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi119-79499-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi119-79477-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi119-79477-27"}]': finished 2024-09-10T19:05:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/714028524' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi119-79801-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3566911626' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi119-79499-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:05:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[40674]: osdmap e151: 8 total, 8 up, 8 in 2024-09-10T19:05:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13"}]: dispatch 2024-09-10T19:05:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13"}]: dispatch 2024-09-10T19:05:31.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:31.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[29577]: osdmap e150: 8 total, 8 up, 8 in 2024-09-10T19:05:31.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/714028524' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi119-79801-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:31.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3566911626' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi119-79499-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:31.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi119-79477-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi119-79477-27"}]': finished 2024-09-10T19:05:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/714028524' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi119-79801-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3566911626' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi119-79499-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:05:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[29577]: osdmap e151: 8 total, 8 up, 8 in 2024-09-10T19:05:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13"}]: dispatch 2024-09-10T19:05:31.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13"}]: dispatch 2024-09-10T19:05:33.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:32 smithi194 ceph-mon[32464]: pgmap v189: 332 pgs: 64 unknown, 268 active+clean; 606 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 8.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T19:05:33.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:32 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13"}]': finished 2024-09-10T19:05:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:32 smithi194 ceph-mon[32464]: osdmap e152: 8 total, 8 up, 8 in 2024-09-10T19:05:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:33.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:05:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:05:32] "GET /metrics HTTP/1.1" 200 240444 "" "Prometheus/2.43.0" 2024-09-10T19:05:33.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:32 smithi119 ceph-mon[40674]: pgmap v189: 332 pgs: 64 unknown, 268 active+clean; 606 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 8.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T19:05:33.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:33.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:32 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:33.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13"}]': finished 2024-09-10T19:05:33.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:32 smithi119 ceph-mon[40674]: osdmap e152: 8 total, 8 up, 8 in 2024-09-10T19:05:33.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:33.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:32 smithi119 ceph-mon[29577]: pgmap v189: 332 pgs: 64 unknown, 268 active+clean; 606 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.2 KiB/s rd, 8.0 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-09-10T19:05:33.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:33.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:32 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:33.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-13"}]': finished 2024-09-10T19:05:33.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:32 smithi119 ceph-mon[29577]: osdmap e152: 8 total, 8 up, 8 in 2024-09-10T19:05:33.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:33 smithi194 ceph-mon[32464]: pgmap v192: 275 pgs: 275 active+clean; 609 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-09-10T19:05:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-09-10T19:05:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-09-10T19:05:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:33 smithi194 ceph-mon[32464]: osdmap e153: 8 total, 8 up, 8 in 2024-09-10T19:05:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2927763915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi119-79499-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2623229204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi119-79801-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi119-79801-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[29577]: pgmap v192: 275 pgs: 275 active+clean; 609 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[29577]: osdmap e153: 8 total, 8 up, 8 in 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2927763915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi119-79499-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2623229204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi119-79801-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi119-79801-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:34.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[40674]: pgmap v192: 275 pgs: 275 active+clean; 609 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-09-10T19:05:34.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-09-10T19:05:34.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi119-79624-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-09-10T19:05:34.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[40674]: osdmap e153: 8 total, 8 up, 8 in 2024-09-10T19:05:34.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2927763915' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi119-79499-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:34.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:34.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2623229204' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi119-79801-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:34.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:34.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:34.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi119-79801-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:35.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:34 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2927763915' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi119-79499-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]': finished 2024-09-10T19:05:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi119-79801-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:34 smithi194 ceph-mon[32464]: osdmap e154: 8 total, 8 up, 8 in 2024-09-10T19:05:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2927763915' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi119-79499-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]': finished 2024-09-10T19:05:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi119-79801-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[29577]: osdmap e154: 8 total, 8 up, 8 in 2024-09-10T19:05:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2927763915' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi119-79499-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi119-79477-27"}]': finished 2024-09-10T19:05:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi119-79801-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[40674]: osdmap e154: 8 total, 8 up, 8 in 2024-09-10T19:05:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2622013126' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]: dispatch 2024-09-10T19:05:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:35.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3083489958' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-09-10T19:05:36.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: pgmap v195: 331 pgs: 96 unknown, 235 active+clean; 609 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-09-10T19:05:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]': finished 2024-09-10T19:05:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: osdmap e155: 8 total, 8 up, 8 in 2024-09-10T19:05:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi119-79477-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi119-79624-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:36.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:36.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:36.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:36.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: pgmap v195: 331 pgs: 96 unknown, 235 active+clean; 609 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-09-10T19:05:36.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]': finished 2024-09-10T19:05:36.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:36.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: osdmap e155: 8 total, 8 up, 8 in 2024-09-10T19:05:36.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:36.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:36.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:36.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi119-79477-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:36.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:36.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi119-79624-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:36.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: pgmap v195: 331 pgs: 96 unknown, 235 active+clean; 609 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 8.0 KiB/s wr, 1 op/s 2024-09-10T19:05:36.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi119-79477-27"}]': finished 2024-09-10T19:05:36.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:36.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: osdmap e155: 8 total, 8 up, 8 in 2024-09-10T19:05:36.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:36.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:36.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:36.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi119-79477-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:36.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:36.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi119-79624-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:36.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:36.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:36.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:36.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:36.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:36.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:38.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi119-79477-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:38.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi119-79624-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:38.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:38.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:37 smithi194 ceph-mon[32464]: osdmap e156: 8 total, 8 up, 8 in 2024-09-10T19:05:38.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi119-79624-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:38.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi119-79477-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:38.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi119-79801-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:38.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3828616104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi119-79499-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:38.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi119-79801-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:38.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:38.244 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi119-79499-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:38.244 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi119-79477-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi119-79624-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[40674]: osdmap e156: 8 total, 8 up, 8 in 2024-09-10T19:05:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi119-79624-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi119-79477-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi119-79801-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3828616104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi119-79499-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi119-79801-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:38.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi119-79499-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:38.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:38.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi119-79477-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:38.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi119-79624-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:38.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:38.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[29577]: osdmap e156: 8 total, 8 up, 8 in 2024-09-10T19:05:38.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi119-79624-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:38.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi119-79477-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:38.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi119-79801-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:38.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3828616104' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi119-79499-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:38.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi119-79801-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:38.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:38.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi119-79499-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:38.319 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:38.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:05:39.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComple: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-10T19:05:39.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-10T19:05:39.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4055 ms) 2024-09-10T19:05:39.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-10T19:05:39.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3923 ms) 2024-09-10T19:05:39.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-10T19:05:39.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4014 ms) 2024-09-10T19:05:39.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11992 ms total) 2024-09-10T19:05:39.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: 2024-09-10T19:05:39.197 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-10T19:05:39.198 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [==========] 20 tests from 5 test suites ran. (97834 ms total) 2024-09-10T19:05:39.198 INFO:tasks.workunit.client.0.smithi119.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-10T19:05:39.245 INFO:tasks.workunit.client.0.smithi119.stdout:expected=0 2024-09-10T19:05:39.245 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:62a1935d:::14:head 2024-09-10T19:05:39.245 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:62a1935d:::14:head expected=14:62a1935d:::14:head 2024-09-10T19:05:39.245 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:62a1935d:::14:head -> 14 2024-09-10T19:05:39.245 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=14 expected=14 2024-09-10T19:05:39.245 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:566253c9:::13:head 2024-09-10T19:05:39.245 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:566253c9:::13:head expected=14:566253c9:::13:head 2024-09-10T19:05:39.245 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:566253c9:::13:head -> 13 2024-09-10T19:05:39.245 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=13 expected=13 2024-09-10T19:05:39.245 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:de5d7c5f:::12:head 2024-09-10T19:05:39.246 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:de5d7c5f:::12:head expected=14:de5d7c5f:::12:head 2024-09-10T19:05:39.246 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:de5d7c5f:::12:head -> 12 2024-09-10T19:05:39.246 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=12 expected=12 2024-09-10T19:05:39.246 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:d83876eb:::4:head 2024-09-10T19:05:39.246 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:d83876eb:::4:head expected=14:d83876eb:::4:head 2024-09-10T19:05:39.246 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:d83876eb:::4:head -> 4 2024-09-10T19:05:39.246 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=4 expected=4 2024-09-10T19:05:39.246 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:cfc208b3:::3:head 2024-09-10T19:05:39.246 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:cfc208b3:::3:head expected=14:cfc208b3:::3:head 2024-09-10T19:05:39.246 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:cfc208b3:::3:head -> 3 2024-09-10T19:05:39.246 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=3 expected=3 2024-09-10T19:05:39.246 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:c4fdafeb:::6:head 2024-09-10T19:05:39.246 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:c4fdafeb:::6:head expected=14:c4fdafeb:::6:head 2024-09-10T19:05:39.246 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:c4fdafeb:::6:head -> 6 2024-09-10T19:05:39.247 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=6 expected=6 2024-09-10T19:05:39.247 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:bd63b0f1:::8:head 2024-09-10T19:05:39.247 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:bd63b0f1:::8:head expected=14:bd63b0f1:::8:head 2024-09-10T19:05:39.247 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:bd63b0f1:::8:head -> 8 2024-09-10T19:05:39.247 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=8 expected=8 2024-09-10T19:05:39.247 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : seek to 14:52ea6a34:::10:head 2024-09-10T19:05:39.247 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : cursor()=14:52ea6a34:::10:head expected=14:52ea6a34:::10:head 2024-09-10T19:05:39.247 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: > 14:52ea6a34:::10:head -> 10 2024-09-10T19:05:39.247 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: : entry=10 expected=10 2024-09-10T19:05:39.247 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (172 ms) 2024-09-10T19:05:39.247 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-10T19:05:39.247 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (76838 ms) 2024-09-10T19:05:39.247 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-10T19:05:39.248 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: split 0/5 -> MIN 14:33333333::::head 2024-09-10T19:05:39.248 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: split 1/5 -> 14:33333333::::head 14:66666666::::head 2024-09-10T19:05:39.248 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: split 2/5 -> 14:66666666::::head 14:99999999::::head 2024-09-10T19:05:39.248 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: split 3/5 -> 14:99999999::::head 14:cccccccc::::head 2024-09-10T19:05:39.248 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: split 4/5 -> 14:cccccccc::::head MAX 2024-09-10T19:05:39.248 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (9856 ms) 2024-09-10T19:05:39.248 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [----------] 7 tests from LibRadosList (87118 ms total) 2024-09-10T19:05:39.248 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 2024-09-10T19:05:39.248 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-10T19:05:39.248 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-10T19:05:39.248 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1024 ms) 2024-09-10T19:05:39.248 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-10T19:05:39.248 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: myset foo1,foo2,foo3 2024-09-10T19:05:39.248 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo1 2024-09-10T19:05:39.249 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo2 2024-09-10T19:05:39.249 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo3 2024-09-10T19:05:39.249 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: myset foo1,foo4,foo5 2024-09-10T19:05:39.249 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo4 2024-09-10T19:05:39.249 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo5 2024-09-10T19:05:39.249 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo1 2024-09-10T19:05:39.249 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: myset foo6,foo7 2024-09-10T19:05:39.249 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo7 2024-09-10T19:05:39.249 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: foo6 2024-09-10T19:05:39.249 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-10T19:05:39.249 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: ns1:foo4 2024-09-10T19:05:39.249 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: ns1:foo5 2024-09-10T19:05:39.249 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: ns2:foo7 2024-09-10T19:05:39.249 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: ns2:foo6 2024-09-10T19:05:39.250 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: ns1:foo1 2024-09-10T19:05:39.250 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: :foo1 2024-09-10T19:05:39.250 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: :foo2 2024-09-10T19:05:39.250 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: :foo3 2024-09-10T19:05:39.250 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (28 ms) 2024-09-10T19:05:39.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: pgmap v198: 268 pgs: 64 unknown, 204 active+clean; 607 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:39.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi119-79499-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:39.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:39.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: osdmap e157: 8 total, 8 up, 8 in 2024-09-10T19:05:39.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:39.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-15"}]: dispatch 2024-09-10T19:05:39.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:39.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-15"}]: dispatch 2024-09-10T19:05:39.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi119-79624-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi119-79624-2"}]': finished 2024-09-10T19:05:39.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi119-79477-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi119-79477-28"}]': finished 2024-09-10T19:05:39.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi119-79801-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi119-79801-36"}]': finished 2024-09-10T19:05:39.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]': finished 2024-09-10T19:05:39.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-15"}]': finished 2024-09-10T19:05:39.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: osdmap e158: 8 total, 8 up, 8 in 2024-09-10T19:05:39.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-15", "mode": "writeback"}]: dispatch 2024-09-10T19:05:39.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:39.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-15", "mode": "writeback"}]: dispatch 2024-09-10T19:05:39.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:39.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:38 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:39.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: pgmap v198: 268 pgs: 64 unknown, 204 active+clean; 607 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:39.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi119-79499-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:39.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:39.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: osdmap e157: 8 total, 8 up, 8 in 2024-09-10T19:05:39.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:39.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-15"}]: dispatch 2024-09-10T19:05:39.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:39.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-15"}]: dispatch 2024-09-10T19:05:39.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi119-79624-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi119-79624-2"}]': finished 2024-09-10T19:05:39.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi119-79477-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi119-79477-28"}]': finished 2024-09-10T19:05:39.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi119-79801-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi119-79801-36"}]': finished 2024-09-10T19:05:39.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]': finished 2024-09-10T19:05:39.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-15"}]': finished 2024-09-10T19:05:39.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: osdmap e158: 8 total, 8 up, 8 in 2024-09-10T19:05:39.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-15", "mode": "writeback"}]: dispatch 2024-09-10T19:05:39.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:39.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-15", "mode": "writeback"}]: dispatch 2024-09-10T19:05:39.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:39.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:39.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: pgmap v198: 268 pgs: 64 unknown, 204 active+clean; 607 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:39.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi119-79499-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:39.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:39.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: osdmap e157: 8 total, 8 up, 8 in 2024-09-10T19:05:39.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:39.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-15"}]: dispatch 2024-09-10T19:05:39.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:39.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-15"}]: dispatch 2024-09-10T19:05:39.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi119-79624-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi119-79624-2"}]': finished 2024-09-10T19:05:39.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi119-79477-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi119-79477-28"}]': finished 2024-09-10T19:05:39.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi119-79801-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi119-79801-36"}]': finished 2024-09-10T19:05:39.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]': finished 2024-09-10T19:05:39.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-15"}]': finished 2024-09-10T19:05:39.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: osdmap e158: 8 total, 8 up, 8 in 2024-09-10T19:05:39.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-15", "mode": "writeback"}]: dispatch 2024-09-10T19:05:39.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1387808357' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:39.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-15", "mode": "writeback"}]: dispatch 2024-09-10T19:05:39.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]: dispatch 2024-09-10T19:05:39.320 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:38 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:40.268 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:05:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:05:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:05:40.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:39 smithi194 ceph-mon[32464]: pgmap v201: 252 pgs: 24 unknown, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:40.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:39 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:40.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-15", "mode": "writeback"}]': finished 2024-09-10T19:05:40.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]': finished 2024-09-10T19:05:40.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:39 smithi194 ceph-mon[32464]: osdmap e159: 8 total, 8 up, 8 in 2024-09-10T19:05:40.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:39 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3387893646' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi119-79499-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:40.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi119-79499-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:40.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:39 smithi119 ceph-mon[29577]: pgmap v201: 252 pgs: 24 unknown, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:40.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:39 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:40.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-15", "mode": "writeback"}]': finished 2024-09-10T19:05:40.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]': finished 2024-09-10T19:05:40.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:39 smithi119 ceph-mon[29577]: osdmap e159: 8 total, 8 up, 8 in 2024-09-10T19:05:40.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:39 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3387893646' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi119-79499-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:40.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi119-79499-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:40.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:39 smithi119 ceph-mon[40674]: pgmap v201: 252 pgs: 24 unknown, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:40.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:39 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:40.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-15", "mode": "writeback"}]': finished 2024-09-10T19:05:40.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi119-79837-20"}]': finished 2024-09-10T19:05:40.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:39 smithi119 ceph-mon[40674]: osdmap e159: 8 total, 8 up, 8 in 2024-09-10T19:05:40.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:39 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3387893646' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi119-79499-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:40.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi119-79499-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:41.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:40 smithi194 ceph-mon[32464]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:41.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:40 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi119-79499-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:41.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:40 smithi194 ceph-mon[32464]: osdmap e160: 8 total, 8 up, 8 in 2024-09-10T19:05:41.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:41.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:41.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:41.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:40 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:41.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:41.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:40 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:41.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[29577]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:41.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi119-79499-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:41.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[29577]: osdmap e160: 8 total, 8 up, 8 in 2024-09-10T19:05:41.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:41.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:41.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:41.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:41.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:41.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:41.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[40674]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:41.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi119-79499-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:41.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[40674]: osdmap e160: 8 total, 8 up, 8 in 2024-09-10T19:05:41.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:41.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:41.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:41.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:41.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:41.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:40 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:42.215 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ RUN ] LibRadosListEC.List7111 ms total) 2024-09-10T19:05:42.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: 2024-09-10T19:05:42.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-10T19:05:42.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-10T19:05:42.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (86 ms) 2024-09-10T19:05:42.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (86 ms total) 2024-09-10T19:05:42.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: 2024-09-10T19:05:42.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-10T19:05:42.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-10T19:05:42.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (43 ms) 2024-09-10T19:05:42.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-10T19:05:42.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (6 ms) 2024-09-10T19:05:42.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (50 ms total) 2024-09-10T19:05:42.216 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: 2024-09-10T19:05:42.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-10T19:05:42.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-10T19:05:42.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (28 ms) 2024-09-10T19:05:42.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-10T19:05:42.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (5 ms) 2024-09-10T19:05:42.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (33 ms total) 2024-09-10T19:05:42.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: 2024-09-10T19:05:42.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-10T19:05:42.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-10T19:05:42.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (34 ms) 2024-09-10T19:05:42.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-10T19:05:42.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (4 ms) 2024-09-10T19:05:42.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (38 ms total) 2024-09-10T19:05:42.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: 2024-09-10T19:05:42.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-10T19:05:42.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-10T19:05:42.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1026 ms) 2024-09-10T19:05:42.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1026 ms total) 2024-09-10T19:05:42.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: 2024-09-10T19:05:42.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-10T19:05:42.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (100844 ms total) 2024-09-10T19:05:42.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-10T19:05:42.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: pgmap v204: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi119-79477-28"}]': finished 2024-09-10T19:05:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi119-79624-2"}]': finished 2024-09-10T19:05:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]': finished 2024-09-10T19:05:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:05:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: osdmap e161: 8 total, 8 up, 8 in 2024-09-10T19:05:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15"}]: dispatch 2024-09-10T19:05:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15"}]: dispatch 2024-09-10T19:05:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:05:42.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:05:42.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:05:42.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:42 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:05:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: pgmap v204: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi119-79477-28"}]': finished 2024-09-10T19:05:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi119-79624-2"}]': finished 2024-09-10T19:05:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]': finished 2024-09-10T19:05:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:05:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: osdmap e161: 8 total, 8 up, 8 in 2024-09-10T19:05:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15"}]: dispatch 2024-09-10T19:05:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15"}]: dispatch 2024-09-10T19:05:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:05:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:05:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:05:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:05:42.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: pgmap v204: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:42.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi119-79477-28"}]': finished 2024-09-10T19:05:42.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi119-79624-2"}]': finished 2024-09-10T19:05:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi119-79801-36"}]': finished 2024-09-10T19:05:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:05:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: osdmap e161: 8 total, 8 up, 8 in 2024-09-10T19:05:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15"}]: dispatch 2024-09-10T19:05:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi119-79477-28"}]: dispatch 2024-09-10T19:05:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi119-79624-2"}]: dispatch 2024-09-10T19:05:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3812935247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15"}]: dispatch 2024-09-10T19:05:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]: dispatch 2024-09-10T19:05:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:05:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:05:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:05:42.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:05:43.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:05:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:05:42] "GET /metrics HTTP/1.1" 200 240444 "" "Prometheus/2.43.0" 2024-09-10T19:05:43.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:43 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi119-79477-28"}]': finished 2024-09-10T19:05:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi119-79624-2"}]': finished 2024-09-10T19:05:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15"}]': finished 2024-09-10T19:05:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]': finished 2024-09-10T19:05:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:43 smithi194 ceph-mon[32464]: osdmap e162: 8 total, 8 up, 8 in 2024-09-10T19:05:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3353409661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi119-79499-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi119-79499-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi119-79477-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi119-79477-28"}]': finished 2024-09-10T19:05:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi119-79624-2"}]': finished 2024-09-10T19:05:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15"}]': finished 2024-09-10T19:05:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]': finished 2024-09-10T19:05:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[29577]: osdmap e162: 8 total, 8 up, 8 in 2024-09-10T19:05:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3353409661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi119-79499-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi119-79499-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi119-79477-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2628595480' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi119-79477-28"}]': finished 2024-09-10T19:05:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/574195359' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi119-79624-2"}]': finished 2024-09-10T19:05:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-15"}]': finished 2024-09-10T19:05:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi119-79801-36"}]': finished 2024-09-10T19:05:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[40674]: osdmap e162: 8 total, 8 up, 8 in 2024-09-10T19:05:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3353409661' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi119-79499-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi119-79499-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi119-79477-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:44.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:44 smithi194 ceph-mon[32464]: pgmap v207: 260 pgs: 32 creating+peering, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:05:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi119-79499-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi119-79477-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:44 smithi194 ceph-mon[32464]: osdmap e163: 8 total, 8 up, 8 in 2024-09-10T19:05:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1147995631' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79624-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi119-79477-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79624-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:44.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:44 smithi119 ceph-mon[29577]: pgmap v207: 260 pgs: 32 creating+peering, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:05:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi119-79499-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi119-79477-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:44 smithi119 ceph-mon[29577]: osdmap e163: 8 total, 8 up, 8 in 2024-09-10T19:05:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1147995631' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79624-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi119-79477-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79624-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:44 smithi119 ceph-mon[40674]: pgmap v207: 260 pgs: 32 creating+peering, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:05:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi119-79499-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:44.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi119-79477-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:44.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:44 smithi119 ceph-mon[40674]: osdmap e163: 8 total, 8 up, 8 in 2024-09-10T19:05:44.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1147995631' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79624-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:44.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi119-79477-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:44.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79624-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:45.217 INFO:tasks.workunit.client.0.smithi119.stdout:ObjectsStart 2024-09-10T19:05:45.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 1 0 2024-09-10T19:05:45.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 10 0 2024-09-10T19:05:45.217 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 13 0 2024-09-10T19:05:45.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 7 0 2024-09-10T19:05:45.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 14 0 2024-09-10T19:05:45.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 0 0 2024-09-10T19:05:45.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 15 0 2024-09-10T19:05:45.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 11 0 2024-09-10T19:05:45.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 5 0 2024-09-10T19:05:45.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 8 0 2024-09-10T19:05:45.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 6 0 2024-09-10T19:05:45.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 3 0 2024-09-10T19:05:45.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 4 0 2024-09-10T19:05:45.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 12 0 2024-09-10T19:05:45.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 9 0 2024-09-10T19:05:45.218 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 2 0 2024-09-10T19:05:45.219 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-10T19:05:45.219 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (41 ms) 2024-09-10T19:05:45.219 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [----------] 3 tests from LibRadosListEC (1093 ms total) 2024-09-10T19:05:45.219 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 2024-09-10T19:05:45.219 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-10T19:05:45.219 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-10T19:05:45.219 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3007 ms) 2024-09-10T19:05:45.219 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [----------] 1 test from LibRadosListNP (3007 ms total) 2024-09-10T19:05:45.219 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: 2024-09-10T19:05:45.219 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [----------] Global test environment tear-down 2024-09-10T19:05:45.219 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [==========] 11 tests from 3 test suites ran. (103904 ms total) 2024-09-10T19:05:45.219 INFO:tasks.workunit.client.0.smithi119.stdout: api_list: [ PASSED ] 11 tests. 2024-09-10T19:05:45.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:45 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79624-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:45 smithi194 ceph-mon[32464]: osdmap e164: 8 total, 8 up, 8 in 2024-09-10T19:05:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1147995631' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi119-79624-3","pool2":"test-rados-api-smithi119-79624-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-09-10T19:05:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:45 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:45 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi119-79624-3","pool2":"test-rados-api-smithi119-79624-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-09-10T19:05:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi119-79477-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi119-79477-29"}]': finished 2024-09-10T19:05:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:45 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:45 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi119-79624-3","pool2":"test-rados-api-smithi119-79624-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-09-10T19:05:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:45 smithi194 ceph-mon[32464]: osdmap e165: 8 total, 8 up, 8 in 2024-09-10T19:05:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4126348055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi119-79499-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:45 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi119-79499-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:45.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:45.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:45 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79624-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[29577]: osdmap e164: 8 total, 8 up, 8 in 2024-09-10T19:05:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1147995631' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi119-79624-3","pool2":"test-rados-api-smithi119-79624-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-09-10T19:05:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi119-79624-3","pool2":"test-rados-api-smithi119-79624-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-09-10T19:05:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi119-79477-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi119-79477-29"}]': finished 2024-09-10T19:05:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:45.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi119-79624-3","pool2":"test-rados-api-smithi119-79624-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-09-10T19:05:45.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[29577]: osdmap e165: 8 total, 8 up, 8 in 2024-09-10T19:05:45.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4126348055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi119-79499-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:45.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi119-79499-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:45.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:45.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:45.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79624-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:45.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[40674]: osdmap e164: 8 total, 8 up, 8 in 2024-09-10T19:05:45.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:45.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1147995631' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi119-79624-3","pool2":"test-rados-api-smithi119-79624-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-09-10T19:05:45.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:45.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi119-79624-3","pool2":"test-rados-api-smithi119-79624-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-09-10T19:05:45.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi119-79477-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi119-79477-29"}]': finished 2024-09-10T19:05:45.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:45.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi119-79624-3","pool2":"test-rados-api-smithi119-79624-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-09-10T19:05:45.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[40674]: osdmap e165: 8 total, 8 up, 8 in 2024-09-10T19:05:45.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4126348055' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi119-79499-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:45.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi119-79499-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:45.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:45.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:45 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:46.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:46 smithi194 ceph-mon[32464]: pgmap v210: 260 pgs: 64 unknown, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi119-79499-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:46 smithi194 ceph-mon[32464]: osdmap e166: 8 total, 8 up, 8 in 2024-09-10T19:05:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-17"}]: dispatch 2024-09-10T19:05:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-17"}]: dispatch 2024-09-10T19:05:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:46 smithi119 ceph-mon[29577]: pgmap v210: 260 pgs: 64 unknown, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi119-79499-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:46 smithi119 ceph-mon[29577]: osdmap e166: 8 total, 8 up, 8 in 2024-09-10T19:05:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-17"}]: dispatch 2024-09-10T19:05:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-17"}]: dispatch 2024-09-10T19:05:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:46 smithi119 ceph-mon[40674]: pgmap v210: 260 pgs: 64 unknown, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:05:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi119-79499-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:46.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:46.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:46 smithi119 ceph-mon[40674]: osdmap e166: 8 total, 8 up, 8 in 2024-09-10T19:05:46.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-17"}]: dispatch 2024-09-10T19:05:46.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-17"}]: dispatch 2024-09-10T19:05:47.183 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-09-10T19:05:47.183 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-10T19:05:47.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-10T19:05:47.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-10T19:05:47.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-10T19:05:47.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (6871 ms) 2024-09-10T19:05:47.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-10T19:05:47.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (21095 ms) 2024-09-10T19:05:47.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-10T19:05:47.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6112 ms) 2024-09-10T19:05:47.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-10T19:05:47.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2860 ms) 2024-09-10T19:05:47.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-10T19:05:47.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (2992 ms) 2024-09-10T19:05:47.184 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-10T19:05:47.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (2992 ms) 2024-09-10T19:05:47.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-10T19:05:47.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (3244 ms) 2024-09-10T19:05:47.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-10T19:05:47.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (2761 ms) 2024-09-10T19:05:47.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-10T19:05:47.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2994 ms) 2024-09-10T19:05:47.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-10T19:05:47.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3009 ms) 2024-09-10T19:05:47.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-10T19:05:47.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2996 ms) 2024-09-10T19:05:47.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-10T19:05:47.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3006 ms) 2024-09-10T19:05:47.185 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-10T19:05:47.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3007 ms) 2024-09-10T19:05:47.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-10T19:05:47.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3273 ms) 2024-09-10T19:05:47.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-10T19:05:47.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (2765 ms) 2024-09-10T19:05:47.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-10T19:05:47.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3004 ms) 2024-09-10T19:05:47.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-10T19:05:47.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (2950 ms) 2024-09-10T19:05:47.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-10T19:05:47.186 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3002 ms) 2024-09-10T19:05:47.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-10T19:05:47.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3486 ms) 2024-09-10T19:05:47.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-10T19:05:47.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (2531 ms) 2024-09-10T19:05:47.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-10T19:05:47.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3004 ms) 2024-09-10T19:05:47.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-10T19:05:47.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2974 ms) 2024-09-10T19:05:47.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-10T19:05:47.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3006 ms) 2024-09-10T19:05:47.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-10T19:05:47.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2999 ms) 2024-09-10T19:05:47.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-10T19:05:47.187 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3008 ms) 2024-09-10T19:05:47.188 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-10T19:05:47.188 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3260 ms) 2024-09-10T19:05:47.188 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-10T19:05:48.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:05:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:05:48.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:48 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:05:48.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:48 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:48 smithi194 ceph-mon[32464]: pgmap v213: 268 pgs: 72 unknown, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:48 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-17"}]': finished 2024-09-10T19:05:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:48 smithi194 ceph-mon[32464]: osdmap e167: 8 total, 8 up, 8 in 2024-09-10T19:05:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-17", "mode": "writeback"}]: dispatch 2024-09-10T19:05:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-17", "mode": "writeback"}]: dispatch 2024-09-10T19:05:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:05:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[29577]: pgmap v213: 268 pgs: 72 unknown, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-17"}]': finished 2024-09-10T19:05:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[29577]: osdmap e167: 8 total, 8 up, 8 in 2024-09-10T19:05:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-17", "mode": "writeback"}]: dispatch 2024-09-10T19:05:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-17", "mode": "writeback"}]: dispatch 2024-09-10T19:05:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:05:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:05:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[40674]: pgmap v213: 268 pgs: 72 unknown, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-17"}]': finished 2024-09-10T19:05:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[40674]: osdmap e167: 8 total, 8 up, 8 in 2024-09-10T19:05:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-17", "mode": "writeback"}]: dispatch 2024-09-10T19:05:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-17", "mode": "writeback"}]: dispatch 2024-09-10T19:05:49.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:49 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi119-79477-29"}]': finished 2024-09-10T19:05:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-17", "mode": "writeback"}]': finished 2024-09-10T19:05:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:49 smithi194 ceph-mon[32464]: osdmap e168: 8 total, 8 up, 8 in 2024-09-10T19:05:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/207412348' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi119-79499-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi119-79499-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:49 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:49.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi119-79477-29"}]': finished 2024-09-10T19:05:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-17", "mode": "writeback"}]': finished 2024-09-10T19:05:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[29577]: osdmap e168: 8 total, 8 up, 8 in 2024-09-10T19:05:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/207412348' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi119-79499-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi119-79499-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi119-79477-29"}]': finished 2024-09-10T19:05:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-17", "mode": "writeback"}]': finished 2024-09-10T19:05:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[40674]: osdmap e168: 8 total, 8 up, 8 in 2024-09-10T19:05:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi119-79477-29"}]: dispatch 2024-09-10T19:05:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/207412348' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi119-79499-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi119-79499-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:49 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:50.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:05:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:05:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:05:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:50 smithi194 ceph-mon[32464]: pgmap v216: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi119-79477-29"}]': finished 2024-09-10T19:05:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:50 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi119-79499-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:50 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:05:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:50 smithi194 ceph-mon[32464]: osdmap e169: 8 total, 8 up, 8 in 2024-09-10T19:05:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17"}]: dispatch 2024-09-10T19:05:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:50 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17"}]: dispatch 2024-09-10T19:05:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:50.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:50.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi119-79477-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:50.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[29577]: pgmap v216: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi119-79477-29"}]': finished 2024-09-10T19:05:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi119-79499-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:05:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[29577]: osdmap e169: 8 total, 8 up, 8 in 2024-09-10T19:05:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17"}]: dispatch 2024-09-10T19:05:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17"}]: dispatch 2024-09-10T19:05:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi119-79477-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:50.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[40674]: pgmap v216: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1769593247' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi119-79477-29"}]': finished 2024-09-10T19:05:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi119-79499-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:05:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[40674]: osdmap e169: 8 total, 8 up, 8 in 2024-09-10T19:05:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17"}]: dispatch 2024-09-10T19:05:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17"}]: dispatch 2024-09-10T19:05:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi119-79477-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:51.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:51 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:51.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17"}]': finished 2024-09-10T19:05:51.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi119-79477-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:51.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:51 smithi194 ceph-mon[32464]: osdmap e170: 8 total, 8 up, 8 in 2024-09-10T19:05:51.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi119-79477-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:51.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:51 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:51.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17"}]': finished 2024-09-10T19:05:51.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi119-79477-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:51.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:51 smithi119 ceph-mon[29577]: osdmap e170: 8 total, 8 up, 8 in 2024-09-10T19:05:51.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi119-79477-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:51.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:51 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:51.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-17"}]': finished 2024-09-10T19:05:51.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi119-79477-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:51.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:51 smithi119 ceph-mon[40674]: osdmap e170: 8 total, 8 up, 8 in 2024-09-10T19:05:51.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi119-79477-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:52.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:52 smithi194 ceph-mon[32464]: pgmap v219: 228 pgs: 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:52.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:52 smithi194 ceph-mon[32464]: osdmap e171: 8 total, 8 up, 8 in 2024-09-10T19:05:52.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3844491358' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi119-79499-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:52.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi119-79499-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:52.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:52 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:52.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:52 smithi119 ceph-mon[29577]: pgmap v219: 228 pgs: 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:52.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:52 smithi119 ceph-mon[29577]: osdmap e171: 8 total, 8 up, 8 in 2024-09-10T19:05:52.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3844491358' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi119-79499-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:52.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi119-79499-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:52.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:52 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:52.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:52 smithi119 ceph-mon[40674]: pgmap v219: 228 pgs: 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:52.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:52 smithi119 ceph-mon[40674]: osdmap e171: 8 total, 8 up, 8 in 2024-09-10T19:05:52.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3844491358' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi119-79499-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:52.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi119-79499-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:52.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:52 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:53.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:05:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:05:52] "GET /metrics HTTP/1.1" 200 233132 "" "Prometheus/2.43.0" 2024-09-10T19:05:53.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi119-79477-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi119-79477-30"}]': finished 2024-09-10T19:05:53.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi119-79499-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:53.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:53 smithi194 ceph-mon[32464]: osdmap e172: 8 total, 8 up, 8 in 2024-09-10T19:05:53.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:53.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:53.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi119-79477-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi119-79477-30"}]': finished 2024-09-10T19:05:53.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi119-79499-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:53.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:53 smithi119 ceph-mon[29577]: osdmap e172: 8 total, 8 up, 8 in 2024-09-10T19:05:53.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:53.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:53.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi119-79477-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi119-79477-30"}]': finished 2024-09-10T19:05:53.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi119-79499-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:53.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:53 smithi119 ceph-mon[40674]: osdmap e172: 8 total, 8 up, 8 in 2024-09-10T19:05:53.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:53.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:54.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:54 smithi194 ceph-mon[32464]: pgmap v222: 268 pgs: 40 creating+peering, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:05:54.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:54 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:54.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:54 smithi194 ceph-mon[32464]: osdmap e173: 8 total, 8 up, 8 in 2024-09-10T19:05:54.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:54 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:54.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:54 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:54.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:54 smithi119 ceph-mon[29577]: pgmap v222: 268 pgs: 40 creating+peering, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:05:54.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:54 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:54.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:54 smithi119 ceph-mon[29577]: osdmap e173: 8 total, 8 up, 8 in 2024-09-10T19:05:54.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:54.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:54 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:54.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:54 smithi119 ceph-mon[40674]: pgmap v222: 268 pgs: 40 creating+peering, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:05:54.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:54 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:54.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:54 smithi119 ceph-mon[40674]: osdmap e173: 8 total, 8 up, 8 in 2024-09-10T19:05:54.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:54 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:54.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:54 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:05:55.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:55 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:55.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:55 smithi194 ceph-mon[32464]: osdmap e174: 8 total, 8 up, 8 in 2024-09-10T19:05:55.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:55.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-19"}]: dispatch 2024-09-10T19:05:55.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:55 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-19"}]: dispatch 2024-09-10T19:05:55.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2810341646' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi119-79499-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:55.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:55 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:55 smithi119 ceph-mon[29577]: osdmap e174: 8 total, 8 up, 8 in 2024-09-10T19:05:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-19"}]: dispatch 2024-09-10T19:05:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:55 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-19"}]: dispatch 2024-09-10T19:05:55.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2810341646' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi119-79499-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:55.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:55 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:05:55.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:55 smithi119 ceph-mon[40674]: osdmap e174: 8 total, 8 up, 8 in 2024-09-10T19:05:55.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:55.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-19"}]: dispatch 2024-09-10T19:05:55.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:55 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-19"}]: dispatch 2024-09-10T19:05:55.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2810341646' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi119-79499-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:56.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:56 smithi194 ceph-mon[32464]: pgmap v225: 260 pgs: 32 unknown, 32 creating+peering, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:56.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi119-79477-30"}]': finished 2024-09-10T19:05:56.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:56 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-19"}]': finished 2024-09-10T19:05:56.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2810341646' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi119-79499-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:56.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:56 smithi194 ceph-mon[32464]: osdmap e175: 8 total, 8 up, 8 in 2024-09-10T19:05:56.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-19", "mode": "writeback"}]: dispatch 2024-09-10T19:05:56.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:56.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:56 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-19", "mode": "writeback"}]: dispatch 2024-09-10T19:05:56.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[29577]: pgmap v225: 260 pgs: 32 unknown, 32 creating+peering, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi119-79477-30"}]': finished 2024-09-10T19:05:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-19"}]': finished 2024-09-10T19:05:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2810341646' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi119-79499-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[29577]: osdmap e175: 8 total, 8 up, 8 in 2024-09-10T19:05:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-19", "mode": "writeback"}]: dispatch 2024-09-10T19:05:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:56.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-19", "mode": "writeback"}]: dispatch 2024-09-10T19:05:56.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[40674]: pgmap v225: 260 pgs: 32 unknown, 32 creating+peering, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:05:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi119-79477-30"}]': finished 2024-09-10T19:05:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-19"}]': finished 2024-09-10T19:05:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2810341646' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi119-79499-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[40674]: osdmap e175: 8 total, 8 up, 8 in 2024-09-10T19:05:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-19", "mode": "writeback"}]: dispatch 2024-09-10T19:05:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi119-79477-30"}]: dispatch 2024-09-10T19:05:56.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:56 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-19", "mode": "writeback"}]: dispatch 2024-09-10T19:05:57.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:57 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:57.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi119-79477-30"}]': finished 2024-09-10T19:05:57.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-19", "mode": "writeback"}]': finished 2024-09-10T19:05:57.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:57 smithi194 ceph-mon[32464]: osdmap e176: 8 total, 8 up, 8 in 2024-09-10T19:05:57.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:57.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:57.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:57.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:57.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:57.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:57.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi119-79477-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:57.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi119-79477-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:57.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:57 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi119-79477-30"}]': finished 2024-09-10T19:05:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-19", "mode": "writeback"}]': finished 2024-09-10T19:05:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[29577]: osdmap e176: 8 total, 8 up, 8 in 2024-09-10T19:05:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:57.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:57.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi119-79477-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:57.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi119-79477-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:57.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:57.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:05:57.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2425007952' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi119-79477-30"}]': finished 2024-09-10T19:05:57.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-19", "mode": "writeback"}]': finished 2024-09-10T19:05:57.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[40674]: osdmap e176: 8 total, 8 up, 8 in 2024-09-10T19:05:57.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:57.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:57.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:57.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:05:57.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:57.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:57.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi119-79477-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:57.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi119-79477-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:05:57.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:57 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:05:58.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:05:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:05:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[29577]: pgmap v228: 228 pgs: 32 creating+peering, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:05:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi119-79477-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[29577]: osdmap e177: 8 total, 8 up, 8 in 2024-09-10T19:05:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1777511591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi119-79499-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19"}]: dispatch 2024-09-10T19:05:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi119-79477-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi119-79499-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi119-79477-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:58.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19"}]: dispatch 2024-09-10T19:05:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[40674]: pgmap v228: 228 pgs: 32 creating+peering, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:05:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi119-79477-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[40674]: osdmap e177: 8 total, 8 up, 8 in 2024-09-10T19:05:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1777511591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi119-79499-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19"}]: dispatch 2024-09-10T19:05:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi119-79477-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi119-79499-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:58.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi119-79477-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:58.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19"}]: dispatch 2024-09-10T19:05:59.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:58 smithi194 ceph-mon[32464]: pgmap v228: 228 pgs: 32 creating+peering, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:05:59.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:05:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi119-79477-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:05:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:58 smithi194 ceph-mon[32464]: osdmap e177: 8 total, 8 up, 8 in 2024-09-10T19:05:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1777511591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi119-79499-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19"}]: dispatch 2024-09-10T19:05:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi119-79477-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi119-79499-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:05:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi119-79477-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:05:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19"}]: dispatch 2024-09-10T19:05:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi119-79499-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1777511591' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi119-79499-33", "var": "eio", "val": "true"}]: dispatch 2024-09-10T19:05:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:59.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19"}]': finished 2024-09-10T19:05:59.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[29577]: osdmap e178: 8 total, 8 up, 8 in 2024-09-10T19:05:59.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi119-79499-33", "var": "eio", "val": "true"}]: dispatch 2024-09-10T19:05:59.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi119-79477-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi119-79477-31"}]': finished 2024-09-10T19:05:59.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi119-79499-33", "var": "eio", "val": "true"}]': finished 2024-09-10T19:05:59.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[29577]: osdmap e179: 8 total, 8 up, 8 in 2024-09-10T19:05:59.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:05:59.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi119-79499-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:05:59.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1777511591' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi119-79499-33", "var": "eio", "val": "true"}]: dispatch 2024-09-10T19:05:59.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:05:59.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19"}]': finished 2024-09-10T19:05:59.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[40674]: osdmap e178: 8 total, 8 up, 8 in 2024-09-10T19:05:59.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi119-79499-33", "var": "eio", "val": "true"}]: dispatch 2024-09-10T19:05:59.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi119-79477-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi119-79477-31"}]': finished 2024-09-10T19:05:59.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi119-79499-33", "var": "eio", "val": "true"}]': finished 2024-09-10T19:05:59.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:05:59 smithi119 ceph-mon[40674]: osdmap e179: 8 total, 8 up, 8 in 2024-09-10T19:06:00.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:59 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:00.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi119-79499-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:59 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1777511591' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi119-79499-33", "var": "eio", "val": "true"}]: dispatch 2024-09-10T19:06:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:59 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-19"}]': finished 2024-09-10T19:06:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:59 smithi194 ceph-mon[32464]: osdmap e178: 8 total, 8 up, 8 in 2024-09-10T19:06:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi119-79499-33", "var": "eio", "val": "true"}]: dispatch 2024-09-10T19:06:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi119-79477-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi119-79477-31"}]': finished 2024-09-10T19:06:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "PoolEIOFlag_smithi119-79499-33", "var": "eio", "val": "true"}]': finished 2024-09-10T19:06:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:05:59 smithi194 ceph-mon[32464]: osdmap e179: 8 total, 8 up, 8 in 2024-09-10T19:06:00.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:06:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:06:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:06:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:00 smithi119 ceph-mon[29577]: pgmap v231: 260 pgs: 32 creating+peering, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:06:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:00 smithi119 ceph-mon[29577]: osdmap e180: 8 total, 8 up, 8 in 2024-09-10T19:06:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:00 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:00 smithi119 ceph-mon[40674]: pgmap v231: 260 pgs: 32 creating+peering, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:06:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:00 smithi119 ceph-mon[40674]: osdmap e180: 8 total, 8 up, 8 in 2024-09-10T19:06:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:00 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:01.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:00 smithi194 ceph-mon[32464]: pgmap v231: 260 pgs: 32 creating+peering, 228 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:06:01.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:00 smithi194 ceph-mon[32464]: osdmap e180: 8 total, 8 up, 8 in 2024-09-10T19:06:01.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:01.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:00 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[29577]: pgmap v234: 236 pgs: 40 unknown, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[29577]: osdmap e181: 8 total, 8 up, 8 in 2024-09-10T19:06:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2674816036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi119-79499-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:06:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi119-79499-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:06:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:02.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:02.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[40674]: pgmap v234: 236 pgs: 40 unknown, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:02.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:02.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[40674]: osdmap e181: 8 total, 8 up, 8 in 2024-09-10T19:06:02.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2674816036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi119-79499-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:02.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:06:02.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi119-79499-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:02.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:06:02.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:02.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:02.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:02.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:02 smithi194 ceph-mon[32464]: pgmap v234: 236 pgs: 40 unknown, 196 active+clean; 583 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:02 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:02 smithi194 ceph-mon[32464]: osdmap e181: 8 total, 8 up, 8 in 2024-09-10T19:06:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2674816036' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi119-79499-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:06:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:02 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi119-79499-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:02 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:06:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:02 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:02 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:03.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:06:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:06:02] "GET /metrics HTTP/1.1" 200 230734 "" "Prometheus/2.43.0" 2024-09-10T19:06:03.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi119-79499-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]': finished 2024-09-10T19:06:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[40674]: osdmap e182: 8 total, 8 up, 8 in 2024-09-10T19:06:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-21"}]: dispatch 2024-09-10T19:06:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:06:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:06:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-21"}]: dispatch 2024-09-10T19:06:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi119-79499-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]': finished 2024-09-10T19:06:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[29577]: osdmap e182: 8 total, 8 up, 8 in 2024-09-10T19:06:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-21"}]: dispatch 2024-09-10T19:06:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:06:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:06:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-21"}]: dispatch 2024-09-10T19:06:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi119-79499-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi119-79477-31"}]': finished 2024-09-10T19:06:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:03 smithi194 ceph-mon[32464]: osdmap e182: 8 total, 8 up, 8 in 2024-09-10T19:06:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-21"}]: dispatch 2024-09-10T19:06:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4173935487' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:06:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]: dispatch 2024-09-10T19:06:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-21"}]: dispatch 2024-09-10T19:06:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[29577]: pgmap v237: 260 pgs: 260 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 25 op/s 2024-09-10T19:06:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]': finished 2024-09-10T19:06:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-21"}]': finished 2024-09-10T19:06:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[29577]: osdmap e183: 8 total, 8 up, 8 in 2024-09-10T19:06:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-21", "mode": "writeback"}]: dispatch 2024-09-10T19:06:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-21", "mode": "writeback"}]: dispatch 2024-09-10T19:06:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi119-79477-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[40674]: pgmap v237: 260 pgs: 260 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 25 op/s 2024-09-10T19:06:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]': finished 2024-09-10T19:06:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-21"}]': finished 2024-09-10T19:06:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[40674]: osdmap e183: 8 total, 8 up, 8 in 2024-09-10T19:06:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-21", "mode": "writeback"}]: dispatch 2024-09-10T19:06:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-21", "mode": "writeback"}]: dispatch 2024-09-10T19:06:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi119-79477-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:04.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:04 smithi194 ceph-mon[32464]: pgmap v237: 260 pgs: 260 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 25 op/s 2024-09-10T19:06:04.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi119-79477-31"}]': finished 2024-09-10T19:06:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-21"}]': finished 2024-09-10T19:06:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:04 smithi194 ceph-mon[32464]: osdmap e183: 8 total, 8 up, 8 in 2024-09-10T19:06:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-21", "mode": "writeback"}]: dispatch 2024-09-10T19:06:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-21", "mode": "writeback"}]: dispatch 2024-09-10T19:06:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi119-79477-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:05.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:05.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-21", "mode": "writeback"}]': finished 2024-09-10T19:06:05.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi119-79477-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:05.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[29577]: osdmap e184: 8 total, 8 up, 8 in 2024-09-10T19:06:05.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/413128151' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi119-79499-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:05.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi119-79477-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:05.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi119-79499-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:05.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:05.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:05.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:05.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-21", "mode": "writeback"}]': finished 2024-09-10T19:06:05.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi119-79477-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:05.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[40674]: osdmap e184: 8 total, 8 up, 8 in 2024-09-10T19:06:05.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/413128151' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi119-79499-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:05.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi119-79477-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:05.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi119-79499-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:05.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:05.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:06.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:05 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:06.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-21", "mode": "writeback"}]': finished 2024-09-10T19:06:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi119-79477-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:05 smithi194 ceph-mon[32464]: osdmap e184: 8 total, 8 up, 8 in 2024-09-10T19:06:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/413128151' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi119-79499-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi119-79477-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi119-79499-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[29577]: pgmap v240: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi119-79499-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[29577]: osdmap e185: 8 total, 8 up, 8 in 2024-09-10T19:06:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21"}]: dispatch 2024-09-10T19:06:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21"}]: dispatch 2024-09-10T19:06:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi119-79477-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi119-79477-32"}]': finished 2024-09-10T19:06:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21"}]': finished 2024-09-10T19:06:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[29577]: osdmap e186: 8 total, 8 up, 8 in 2024-09-10T19:06:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[40674]: pgmap v240: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi119-79499-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[40674]: osdmap e185: 8 total, 8 up, 8 in 2024-09-10T19:06:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21"}]: dispatch 2024-09-10T19:06:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21"}]: dispatch 2024-09-10T19:06:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi119-79477-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi119-79477-32"}]': finished 2024-09-10T19:06:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21"}]': finished 2024-09-10T19:06:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:06 smithi119 ceph-mon[40674]: osdmap e186: 8 total, 8 up, 8 in 2024-09-10T19:06:07.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:06 smithi194 ceph-mon[32464]: pgmap v240: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:07.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:06 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi119-79499-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:06 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:06 smithi194 ceph-mon[32464]: osdmap e185: 8 total, 8 up, 8 in 2024-09-10T19:06:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21"}]: dispatch 2024-09-10T19:06:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:06 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21"}]: dispatch 2024-09-10T19:06:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:06 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi119-79477-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi119-79477-32"}]': finished 2024-09-10T19:06:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:06 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-21"}]': finished 2024-09-10T19:06:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:06 smithi194 ceph-mon[32464]: osdmap e186: 8 total, 8 up, 8 in 2024-09-10T19:06:07.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:07 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:07 smithi119 ceph-mon[29577]: osdmap e187: 8 total, 8 up, 8 in 2024-09-10T19:06:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/290577924' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:07 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:07 smithi119 ceph-mon[40674]: osdmap e187: 8 total, 8 up, 8 in 2024-09-10T19:06:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/290577924' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:07 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:07 smithi194 ceph-mon[32464]: osdmap e187: 8 total, 8 up, 8 in 2024-09-10T19:06:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/290577924' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:08.567 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:06:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:06:09.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:08 smithi194 ceph-mon[32464]: pgmap v243: 236 pgs: 8 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:09.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:09.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:08 smithi194 ceph-mon[32464]: osdmap e188: 8 total, 8 up, 8 in 2024-09-10T19:06:09.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:09.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:09.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:09.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:08 smithi119 ceph-mon[40674]: pgmap v243: 236 pgs: 8 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:09.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:09.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:08 smithi119 ceph-mon[40674]: osdmap e188: 8 total, 8 up, 8 in 2024-09-10T19:06:09.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:09.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:09.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:09.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:08 smithi119 ceph-mon[29577]: pgmap v243: 236 pgs: 8 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:09.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:09.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:08 smithi119 ceph-mon[29577]: osdmap e188: 8 total, 8 up, 8 in 2024-09-10T19:06:09.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:09.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:09.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:10.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:09 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi119-79477-32"}]': finished 2024-09-10T19:06:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:09 smithi194 ceph-mon[32464]: osdmap e189: 8 total, 8 up, 8 in 2024-09-10T19:06:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4283502857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:09 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi119-79477-32"}]': finished 2024-09-10T19:06:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:09 smithi119 ceph-mon[29577]: osdmap e189: 8 total, 8 up, 8 in 2024-09-10T19:06:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4283502857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:10.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:09 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:10.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:10.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi119-79477-32"}]': finished 2024-09-10T19:06:10.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:09 smithi119 ceph-mon[40674]: osdmap e189: 8 total, 8 up, 8 in 2024-09-10T19:06:10.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi119-79477-32"}]: dispatch 2024-09-10T19:06:10.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4283502857' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:10.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:10.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:06:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:06:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:06:11.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:10 smithi194 ceph-mon[32464]: pgmap v246: 260 pgs: 32 creating+peering, 32 unknown, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:11.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi119-79477-32"}]': finished 2024-09-10T19:06:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:10 smithi194 ceph-mon[32464]: osdmap e190: 8 total, 8 up, 8 in 2024-09-10T19:06:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi119-79477-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:11.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[29577]: pgmap v246: 260 pgs: 32 creating+peering, 32 unknown, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:11.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi119-79477-32"}]': finished 2024-09-10T19:06:11.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:11.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[29577]: osdmap e190: 8 total, 8 up, 8 in 2024-09-10T19:06:11.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:11.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:11.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:11.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:11.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi119-79477-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:11.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[40674]: pgmap v246: 260 pgs: 32 creating+peering, 32 unknown, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:11.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/993227037' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi119-79477-32"}]': finished 2024-09-10T19:06:11.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:11.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[40674]: osdmap e190: 8 total, 8 up, 8 in 2024-09-10T19:06:11.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:11.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:11.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:11.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:11.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi119-79477-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:12.566 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[40674]: pgmap v249: 292 pgs: 32 creating+peering, 64 unknown, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:12.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:12.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi119-79477-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:12.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[40674]: osdmap e191: 8 total, 8 up, 8 in 2024-09-10T19:06:12.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi119-79477-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:12.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-23"}]: dispatch 2024-09-10T19:06:12.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3767606389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:12.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:12.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-23"}]: dispatch 2024-09-10T19:06:12.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[29577]: pgmap v249: 292 pgs: 32 creating+peering, 64 unknown, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi119-79477-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[29577]: osdmap e191: 8 total, 8 up, 8 in 2024-09-10T19:06:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi119-79477-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-23"}]: dispatch 2024-09-10T19:06:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3767606389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-23"}]: dispatch 2024-09-10T19:06:12.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:12.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:12 smithi194 ceph-mon[32464]: pgmap v249: 292 pgs: 32 creating+peering, 64 unknown, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:06:12.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi119-79477-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:12 smithi194 ceph-mon[32464]: osdmap e191: 8 total, 8 up, 8 in 2024-09-10T19:06:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi119-79477-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-23"}]: dispatch 2024-09-10T19:06:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3767606389' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-23"}]: dispatch 2024-09-10T19:06:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:12 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:13.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:06:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:06:12] "GET /metrics HTTP/1.1" 200 230734 "" "Prometheus/2.43.0" 2024-09-10T19:06:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-23"}]': finished 2024-09-10T19:06:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:13 smithi119 ceph-mon[29577]: osdmap e192: 8 total, 8 up, 8 in 2024-09-10T19:06:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-23", "mode": "writeback"}]: dispatch 2024-09-10T19:06:13.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-23", "mode": "writeback"}]: dispatch 2024-09-10T19:06:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-23"}]': finished 2024-09-10T19:06:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:13 smithi119 ceph-mon[40674]: osdmap e192: 8 total, 8 up, 8 in 2024-09-10T19:06:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-23", "mode": "writeback"}]: dispatch 2024-09-10T19:06:13.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-23", "mode": "writeback"}]: dispatch 2024-09-10T19:06:13.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi119-79499-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-23"}]': finished 2024-09-10T19:06:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:13 smithi194 ceph-mon[32464]: osdmap e192: 8 total, 8 up, 8 in 2024-09-10T19:06:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-23", "mode": "writeback"}]: dispatch 2024-09-10T19:06:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-23", "mode": "writeback"}]: dispatch 2024-09-10T19:06:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:14 smithi119 ceph-mon[29577]: pgmap v252: 324 pgs: 324 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T19:06:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:14 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi119-79477-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi119-79477-33"}]': finished 2024-09-10T19:06:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-23", "mode": "writeback"}]': finished 2024-09-10T19:06:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:14 smithi119 ceph-mon[29577]: osdmap e193: 8 total, 8 up, 8 in 2024-09-10T19:06:14.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:14 smithi119 ceph-mon[40674]: pgmap v252: 324 pgs: 324 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T19:06:14.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:14 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:14.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi119-79477-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi119-79477-33"}]': finished 2024-09-10T19:06:14.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-23", "mode": "writeback"}]': finished 2024-09-10T19:06:14.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:14 smithi119 ceph-mon[40674]: osdmap e193: 8 total, 8 up, 8 in 2024-09-10T19:06:14.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:14 smithi194 ceph-mon[32464]: pgmap v252: 324 pgs: 324 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T19:06:14.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:14 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi119-79477-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi119-79477-33"}]': finished 2024-09-10T19:06:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-23", "mode": "writeback"}]': finished 2024-09-10T19:06:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:14 smithi194 ceph-mon[32464]: osdmap e193: 8 total, 8 up, 8 in 2024-09-10T19:06:15.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:15 smithi119 ceph-mon[29577]: osdmap e194: 8 total, 8 up, 8 in 2024-09-10T19:06:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:15 smithi119 ceph-mon[40674]: osdmap e194: 8 total, 8 up, 8 in 2024-09-10T19:06:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:15.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:15.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:15 smithi194 ceph-mon[32464]: osdmap e194: 8 total, 8 up, 8 in 2024-09-10T19:06:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:16.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:16 smithi194 ceph-mon[32464]: pgmap v255: 268 pgs: 8 unknown, 260 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T19:06:16.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:16.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:16 smithi194 ceph-mon[32464]: osdmap e195: 8 total, 8 up, 8 in 2024-09-10T19:06:16.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23"}]: dispatch 2024-09-10T19:06:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23"}]: dispatch 2024-09-10T19:06:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:16 smithi119 ceph-mon[29577]: pgmap v255: 268 pgs: 8 unknown, 260 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T19:06:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:16 smithi119 ceph-mon[29577]: osdmap e195: 8 total, 8 up, 8 in 2024-09-10T19:06:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23"}]: dispatch 2024-09-10T19:06:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23"}]: dispatch 2024-09-10T19:06:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:16 smithi119 ceph-mon[40674]: pgmap v255: 268 pgs: 8 unknown, 260 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 3 op/s 2024-09-10T19:06:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:16 smithi119 ceph-mon[40674]: osdmap e195: 8 total, 8 up, 8 in 2024-09-10T19:06:16.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:16.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23"}]: dispatch 2024-09-10T19:06:16.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23"}]: dispatch 2024-09-10T19:06:18.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:17 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi119-79477-33"}]': finished 2024-09-10T19:06:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23"}]': finished 2024-09-10T19:06:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:17 smithi194 ceph-mon[32464]: osdmap e196: 8 total, 8 up, 8 in 2024-09-10T19:06:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2042953031' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:17 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi119-79477-33"}]': finished 2024-09-10T19:06:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2042953031' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:17 smithi194 ceph-mon[32464]: osdmap e197: 8 total, 8 up, 8 in 2024-09-10T19:06:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi119-79477-33"}]': finished 2024-09-10T19:06:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23"}]': finished 2024-09-10T19:06:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[29577]: osdmap e196: 8 total, 8 up, 8 in 2024-09-10T19:06:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2042953031' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi119-79477-33"}]': finished 2024-09-10T19:06:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2042953031' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:18.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[29577]: osdmap e197: 8 total, 8 up, 8 in 2024-09-10T19:06:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi119-79477-33"}]': finished 2024-09-10T19:06:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-23"}]': finished 2024-09-10T19:06:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[40674]: osdmap e196: 8 total, 8 up, 8 in 2024-09-10T19:06:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi119-79477-33"}]: dispatch 2024-09-10T19:06:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2042953031' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/163195015' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi119-79477-33"}]': finished 2024-09-10T19:06:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2042953031' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:18.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:17 smithi119 ceph-mon[40674]: osdmap e197: 8 total, 8 up, 8 in 2024-09-10T19:06:18.581 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:06:18.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: pgmap v258: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:18.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:18.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:18.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:18.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:18.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi119-79477-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:18.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi119-79477-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:18.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi119-79477-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:18.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: osdmap e198: 8 total, 8 up, 8 in 2024-09-10T19:06:18.970 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi119-79477-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:18.970 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:18.970 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi119-79477-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:18.970 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:18.970 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/447372421' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:18.970 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:18.970 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: pgmap v258: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:18.970 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:18.970 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:18.970 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:18.971 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:18.971 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi119-79477-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:18.971 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi119-79477-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:18.971 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi119-79477-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:18.971 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: osdmap e198: 8 total, 8 up, 8 in 2024-09-10T19:06:18.971 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi119-79477-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:18.971 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:18.971 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi119-79477-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:18.971 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:18.971 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/447372421' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:18.971 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: pgmap v258: 260 pgs: 32 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi119-79477-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi119-79477-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi119-79477-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: osdmap e198: 8 total, 8 up, 8 in 2024-09-10T19:06:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi119-79477-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:19.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:19.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi119-79477-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:19.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:19.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/447372421' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:19.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:20.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:19 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:20.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:19 smithi194 ceph-mon[32464]: osdmap e199: 8 total, 8 up, 8 in 2024-09-10T19:06:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:19 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:19 smithi119 ceph-mon[29577]: osdmap e199: 8 total, 8 up, 8 in 2024-09-10T19:06:20.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:19 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:20.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:20.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:20.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:19 smithi119 ceph-mon[40674]: osdmap e199: 8 total, 8 up, 8 in 2024-09-10T19:06:20.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:06:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:06:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:06:21.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:20 smithi194 ceph-mon[32464]: pgmap v261: 292 pgs: 64 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-10T19:06:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi119-79477-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi119-79477-34"}]': finished 2024-09-10T19:06:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:20 smithi194 ceph-mon[32464]: osdmap e200: 8 total, 8 up, 8 in 2024-09-10T19:06:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3880668171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:20 smithi119 ceph-mon[29577]: pgmap v261: 292 pgs: 64 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-10T19:06:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi119-79477-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi119-79477-34"}]': finished 2024-09-10T19:06:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:20 smithi119 ceph-mon[29577]: osdmap e200: 8 total, 8 up, 8 in 2024-09-10T19:06:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3880668171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:20 smithi119 ceph-mon[40674]: pgmap v261: 292 pgs: 64 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-10T19:06:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi119-79477-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi119-79477-34"}]': finished 2024-09-10T19:06:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:20 smithi119 ceph-mon[40674]: osdmap e200: 8 total, 8 up, 8 in 2024-09-10T19:06:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3880668171' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[29577]: pgmap v264: 332 pgs: 104 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:06:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[29577]: osdmap e201: 8 total, 8 up, 8 in 2024-09-10T19:06:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-25"}]: dispatch 2024-09-10T19:06:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-25"}]: dispatch 2024-09-10T19:06:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-25"}]': finished 2024-09-10T19:06:22.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[29577]: osdmap e202: 8 total, 8 up, 8 in 2024-09-10T19:06:22.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3166681578' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:22.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:22.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-25", "mode": "writeback"}]: dispatch 2024-09-10T19:06:22.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:22.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-25", "mode": "writeback"}]: dispatch 2024-09-10T19:06:22.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[40674]: pgmap v264: 332 pgs: 104 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:06:22.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:22.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:22.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[40674]: osdmap e201: 8 total, 8 up, 8 in 2024-09-10T19:06:22.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-25"}]: dispatch 2024-09-10T19:06:22.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-25"}]: dispatch 2024-09-10T19:06:22.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:22.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-25"}]': finished 2024-09-10T19:06:22.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[40674]: osdmap e202: 8 total, 8 up, 8 in 2024-09-10T19:06:22.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3166681578' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:22.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:22.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-25", "mode": "writeback"}]: dispatch 2024-09-10T19:06:22.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:22.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-25", "mode": "writeback"}]: dispatch 2024-09-10T19:06:22.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:22 smithi194 ceph-mon[32464]: pgmap v264: 332 pgs: 104 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:06:22.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:22 smithi194 ceph-mon[32464]: osdmap e201: 8 total, 8 up, 8 in 2024-09-10T19:06:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-25"}]: dispatch 2024-09-10T19:06:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-25"}]: dispatch 2024-09-10T19:06:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:22 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-25"}]': finished 2024-09-10T19:06:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:22 smithi194 ceph-mon[32464]: osdmap e202: 8 total, 8 up, 8 in 2024-09-10T19:06:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3166681578' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-25", "mode": "writeback"}]: dispatch 2024-09-10T19:06:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-25", "mode": "writeback"}]: dispatch 2024-09-10T19:06:23.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:06:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:06:22] "GET /metrics HTTP/1.1" 200 230763 "" "Prometheus/2.43.0" 2024-09-10T19:06:23.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:23 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:23.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3166681578' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:23.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]': finished 2024-09-10T19:06:23.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-25", "mode": "writeback"}]': finished 2024-09-10T19:06:23.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:23 smithi194 ceph-mon[32464]: osdmap e203: 8 total, 8 up, 8 in 2024-09-10T19:06:23.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:23.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:23.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:23.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3166681578' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]': finished 2024-09-10T19:06:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-25", "mode": "writeback"}]': finished 2024-09-10T19:06:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[29577]: osdmap e203: 8 total, 8 up, 8 in 2024-09-10T19:06:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:23.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:23.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3166681578' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:23.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi119-79477-34"}]': finished 2024-09-10T19:06:23.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-25", "mode": "writeback"}]': finished 2024-09-10T19:06:23.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[40674]: osdmap e203: 8 total, 8 up, 8 in 2024-09-10T19:06:23.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1243624456' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:23.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]: dispatch 2024-09-10T19:06:23.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:23.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:24.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:24 smithi194 ceph-mon[32464]: pgmap v267: 356 pgs: 32 creating+peering, 324 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:06:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]': finished 2024-09-10T19:06:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:24 smithi194 ceph-mon[32464]: osdmap e204: 8 total, 8 up, 8 in 2024-09-10T19:06:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/47578818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25"}]: dispatch 2024-09-10T19:06:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25"}]: dispatch 2024-09-10T19:06:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:24.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi119-79477-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:24.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi119-79477-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[29577]: pgmap v267: 356 pgs: 32 creating+peering, 324 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:06:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]': finished 2024-09-10T19:06:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[29577]: osdmap e204: 8 total, 8 up, 8 in 2024-09-10T19:06:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/47578818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25"}]: dispatch 2024-09-10T19:06:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25"}]: dispatch 2024-09-10T19:06:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi119-79477-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi119-79477-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[40674]: pgmap v267: 356 pgs: 32 creating+peering, 324 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:06:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi119-79477-34"}]': finished 2024-09-10T19:06:24.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:24.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[40674]: osdmap e204: 8 total, 8 up, 8 in 2024-09-10T19:06:24.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/47578818' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:24.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:24.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25"}]: dispatch 2024-09-10T19:06:24.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25"}]: dispatch 2024-09-10T19:06:24.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:24.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:24.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:24.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:24.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi119-79477-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:24.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi119-79477-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:25.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:25 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:25.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:25.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25"}]': finished 2024-09-10T19:06:25.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi119-79477-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:25.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:25 smithi194 ceph-mon[32464]: osdmap e205: 8 total, 8 up, 8 in 2024-09-10T19:06:25.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:25 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi119-79477-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:25.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi119-79477-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:25 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25"}]': finished 2024-09-10T19:06:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi119-79477-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:25 smithi119 ceph-mon[29577]: osdmap e205: 8 total, 8 up, 8 in 2024-09-10T19:06:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:25 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi119-79477-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi119-79477-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:25.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:25 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:25.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi119-79499-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:25.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-25"}]': finished 2024-09-10T19:06:25.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi119-79477-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:25.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:25 smithi119 ceph-mon[40674]: osdmap e205: 8 total, 8 up, 8 in 2024-09-10T19:06:25.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:25 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi119-79477-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:25.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi119-79477-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:27.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:26 smithi194 ceph-mon[32464]: pgmap v270: 388 pgs: 32 unknown, 32 creating+peering, 324 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:06:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:26 smithi194 ceph-mon[32464]: osdmap e206: 8 total, 8 up, 8 in 2024-09-10T19:06:27.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:26 smithi119 ceph-mon[40674]: pgmap v270: 388 pgs: 32 unknown, 32 creating+peering, 324 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:06:27.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:26 smithi119 ceph-mon[40674]: osdmap e206: 8 total, 8 up, 8 in 2024-09-10T19:06:27.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:26 smithi119 ceph-mon[29577]: pgmap v270: 388 pgs: 32 unknown, 32 creating+peering, 324 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:06:27.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:26 smithi119 ceph-mon[29577]: osdmap e206: 8 total, 8 up, 8 in 2024-09-10T19:06:28.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:27 smithi194 ceph-mon[32464]: pgmap v273: 324 pgs: 32 creating+peering, 292 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:28.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:27 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:28.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:27 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi119-79477-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi119-79477-35"}]': finished 2024-09-10T19:06:28.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:27 smithi194 ceph-mon[32464]: osdmap e207: 8 total, 8 up, 8 in 2024-09-10T19:06:28.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:28.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:27 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:28.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:27 smithi119 ceph-mon[29577]: pgmap v273: 324 pgs: 32 creating+peering, 292 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:28.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:27 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:28.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:27 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi119-79477-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi119-79477-35"}]': finished 2024-09-10T19:06:28.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:27 smithi119 ceph-mon[29577]: osdmap e207: 8 total, 8 up, 8 in 2024-09-10T19:06:28.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:28.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:27 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:28.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:27 smithi119 ceph-mon[40674]: pgmap v273: 324 pgs: 32 creating+peering, 292 active+clean; 584 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:28.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:27 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:28.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:27 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi119-79477-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi119-79477-35"}]': finished 2024-09-10T19:06:28.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:27 smithi119 ceph-mon[40674]: osdmap e207: 8 total, 8 up, 8 in 2024-09-10T19:06:28.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:28.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:27 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:28.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:06:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:06:29.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:29.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:29 smithi194 ceph-mon[32464]: osdmap e208: 8 total, 8 up, 8 in 2024-09-10T19:06:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:29 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:29 smithi119 ceph-mon[29577]: osdmap e208: 8 total, 8 up, 8 in 2024-09-10T19:06:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:29 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:29.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:29.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:29 smithi119 ceph-mon[40674]: osdmap e208: 8 total, 8 up, 8 in 2024-09-10T19:06:29.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:29 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:30.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:06:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:06:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:06:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:30 smithi194 ceph-mon[32464]: pgmap v276: 300 pgs: 40 creating+peering, 260 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:06:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:30 smithi194 ceph-mon[32464]: osdmap e209: 8 total, 8 up, 8 in 2024-09-10T19:06:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:30 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:30 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:30 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:30 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:30.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:30 smithi119 ceph-mon[29577]: pgmap v276: 300 pgs: 40 creating+peering, 260 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:06:30.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:30 smithi119 ceph-mon[29577]: osdmap e209: 8 total, 8 up, 8 in 2024-09-10T19:06:30.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:30 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:30.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:30 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:30.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:30 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:30.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:30 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:30.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:30 smithi119 ceph-mon[40674]: pgmap v276: 300 pgs: 40 creating+peering, 260 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:06:30.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:30 smithi119 ceph-mon[40674]: osdmap e209: 8 total, 8 up, 8 in 2024-09-10T19:06:30.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:30 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:30.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:30 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:30.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:30 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:30.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:30 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:31.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]': finished 2024-09-10T19:06:31.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:31.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:31 smithi194 ceph-mon[32464]: osdmap e210: 8 total, 8 up, 8 in 2024-09-10T19:06:31.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-27"}]: dispatch 2024-09-10T19:06:31.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:31.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-27"}]: dispatch 2024-09-10T19:06:31.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:31.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]': finished 2024-09-10T19:06:31.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:31.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:31 smithi119 ceph-mon[29577]: osdmap e210: 8 total, 8 up, 8 in 2024-09-10T19:06:31.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-27"}]: dispatch 2024-09-10T19:06:31.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:31.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-27"}]: dispatch 2024-09-10T19:06:31.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:31.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi119-79477-35"}]': finished 2024-09-10T19:06:31.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:31.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:31 smithi119 ceph-mon[40674]: osdmap e210: 8 total, 8 up, 8 in 2024-09-10T19:06:31.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-27"}]: dispatch 2024-09-10T19:06:31.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4055905263' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:31.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-27"}]: dispatch 2024-09-10T19:06:31.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]: dispatch 2024-09-10T19:06:32.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: pgmap v279: 228 pgs: 32 creating+peering, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:32.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-27"}]': finished 2024-09-10T19:06:32.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]': finished 2024-09-10T19:06:32.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: osdmap e211: 8 total, 8 up, 8 in 2024-09-10T19:06:32.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2536041548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi119-79499-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:32.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-27", "mode": "writeback"}]: dispatch 2024-09-10T19:06:32.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-27", "mode": "writeback"}]: dispatch 2024-09-10T19:06:32.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:32.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:32.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:32.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:32.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:32.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:32.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:32.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:32 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:32.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: pgmap v279: 228 pgs: 32 creating+peering, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:32.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-27"}]': finished 2024-09-10T19:06:32.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]': finished 2024-09-10T19:06:32.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: osdmap e211: 8 total, 8 up, 8 in 2024-09-10T19:06:32.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2536041548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi119-79499-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:32.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-27", "mode": "writeback"}]: dispatch 2024-09-10T19:06:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-27", "mode": "writeback"}]: dispatch 2024-09-10T19:06:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:32.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:32.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: pgmap v279: 228 pgs: 32 creating+peering, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:32.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-27"}]': finished 2024-09-10T19:06:32.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi119-79477-35"}]': finished 2024-09-10T19:06:32.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: osdmap e211: 8 total, 8 up, 8 in 2024-09-10T19:06:32.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2536041548' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi119-79499-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:32.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-27", "mode": "writeback"}]: dispatch 2024-09-10T19:06:32.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-27", "mode": "writeback"}]: dispatch 2024-09-10T19:06:32.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:32.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:32.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:32.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:32.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:32.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:32.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:32.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:33.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:06:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:06:32] "GET /metrics HTTP/1.1" 200 230751 "" "Prometheus/2.43.0" 2024-09-10T19:06:33.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:33 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:33.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2536041548' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi119-79499-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:33.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-27", "mode": "writeback"}]': finished 2024-09-10T19:06:33.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:33.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:33 smithi194 ceph-mon[32464]: osdmap e212: 8 total, 8 up, 8 in 2024-09-10T19:06:33.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi119-79477-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:33.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi119-79477-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:33.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:33 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:33.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2536041548' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi119-79499-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:33.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-27", "mode": "writeback"}]': finished 2024-09-10T19:06:33.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:33.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:33 smithi119 ceph-mon[29577]: osdmap e212: 8 total, 8 up, 8 in 2024-09-10T19:06:33.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi119-79477-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:33.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi119-79477-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:33.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:33 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:33.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2536041548' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi119-79499-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:33.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-27", "mode": "writeback"}]': finished 2024-09-10T19:06:33.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:33.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:33 smithi119 ceph-mon[40674]: osdmap e212: 8 total, 8 up, 8 in 2024-09-10T19:06:33.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi119-79477-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:33.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi119-79477-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:34.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:34 smithi194 ceph-mon[32464]: pgmap v282: 260 pgs: 260 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:06:34.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:34 smithi194 ceph-mon[32464]: osdmap e213: 8 total, 8 up, 8 in 2024-09-10T19:06:34.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:34.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:34.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:34 smithi119 ceph-mon[29577]: pgmap v282: 260 pgs: 260 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:06:34.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:34 smithi119 ceph-mon[29577]: osdmap e213: 8 total, 8 up, 8 in 2024-09-10T19:06:34.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:34.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:34.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:34 smithi119 ceph-mon[40674]: pgmap v282: 260 pgs: 260 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:06:34.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:34 smithi119 ceph-mon[40674]: osdmap e213: 8 total, 8 up, 8 in 2024-09-10T19:06:34.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:34.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:35.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:35 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi119-79477-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi119-79477-36"}]': finished 2024-09-10T19:06:35.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:35 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:35.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:35 smithi194 ceph-mon[32464]: osdmap e214: 8 total, 8 up, 8 in 2024-09-10T19:06:35.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/935786690' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi119-79499-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:35.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27"}]: dispatch 2024-09-10T19:06:35.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:35 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27"}]: dispatch 2024-09-10T19:06:35.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:35 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi119-79477-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi119-79477-36"}]': finished 2024-09-10T19:06:35.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:35 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:35.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:35 smithi119 ceph-mon[29577]: osdmap e214: 8 total, 8 up, 8 in 2024-09-10T19:06:35.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/935786690' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi119-79499-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:35.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27"}]: dispatch 2024-09-10T19:06:35.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:35 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27"}]: dispatch 2024-09-10T19:06:35.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:35 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi119-79477-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi119-79477-36"}]': finished 2024-09-10T19:06:35.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:35 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:35.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:35 smithi119 ceph-mon[40674]: osdmap e214: 8 total, 8 up, 8 in 2024-09-10T19:06:35.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/935786690' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi119-79499-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:35.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27"}]: dispatch 2024-09-10T19:06:35.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:35 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27"}]: dispatch 2024-09-10T19:06:36.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:36 smithi194 ceph-mon[32464]: pgmap v285: 268 pgs: 40 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-09-10T19:06:36.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:36 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:36.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/935786690' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi119-79499-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27"}]': finished 2024-09-10T19:06:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:36 smithi194 ceph-mon[32464]: osdmap e215: 8 total, 8 up, 8 in 2024-09-10T19:06:36.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:36 smithi119 ceph-mon[29577]: pgmap v285: 268 pgs: 40 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-09-10T19:06:36.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:36 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:36.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/935786690' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi119-79499-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:36.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27"}]': finished 2024-09-10T19:06:36.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:36 smithi119 ceph-mon[29577]: osdmap e215: 8 total, 8 up, 8 in 2024-09-10T19:06:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:36 smithi119 ceph-mon[40674]: pgmap v285: 268 pgs: 40 unknown, 228 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-09-10T19:06:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:36 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/935786690' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi119-79499-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-27"}]': finished 2024-09-10T19:06:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:36 smithi119 ceph-mon[40674]: osdmap e215: 8 total, 8 up, 8 in 2024-09-10T19:06:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[29577]: osdmap e216: 8 total, 8 up, 8 in 2024-09-10T19:06:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:37.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:37.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:37.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:37.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[40674]: osdmap e216: 8 total, 8 up, 8 in 2024-09-10T19:06:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:37.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:37 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:37 smithi194 ceph-mon[32464]: osdmap e216: 8 total, 8 up, 8 in 2024-09-10T19:06:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:37.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:37 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:38.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[29577]: pgmap v288: 196 pgs: 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]': finished 2024-09-10T19:06:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[29577]: osdmap e217: 8 total, 8 up, 8 in 2024-09-10T19:06:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:38.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[40674]: pgmap v288: 196 pgs: 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:38.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]': finished 2024-09-10T19:06:38.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:38.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[40674]: osdmap e217: 8 total, 8 up, 8 in 2024-09-10T19:06:38.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:38.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:38.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:38.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:38.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:38.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:38.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:06:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:06:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:38 smithi194 ceph-mon[32464]: pgmap v288: 196 pgs: 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi119-79477-36"}]': finished 2024-09-10T19:06:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:38 smithi194 ceph-mon[32464]: osdmap e217: 8 total, 8 up, 8 in 2024-09-10T19:06:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4240476109' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]: dispatch 2024-09-10T19:06:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]': finished 2024-09-10T19:06:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[29577]: osdmap e218: 8 total, 8 up, 8 in 2024-09-10T19:06:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:39.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi119-79477-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:39.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi119-79477-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:39.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]': finished 2024-09-10T19:06:39.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:39.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:39.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:39.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[40674]: osdmap e218: 8 total, 8 up, 8 in 2024-09-10T19:06:39.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:39.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:39.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:39.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi119-79477-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:39.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi119-79477-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:39.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi119-79477-36"}]': finished 2024-09-10T19:06:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:39 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:39 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:39 smithi194 ceph-mon[32464]: osdmap e218: 8 total, 8 up, 8 in 2024-09-10T19:06:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:39 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:39 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi119-79477-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi119-79477-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:40.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:06:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:06:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:06:40.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:40 smithi194 ceph-mon[32464]: pgmap v291: 228 pgs: 32 creating+peering, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:06:40.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:40 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-46"}]': finished 2024-09-10T19:06:40.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:40 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi119-79477-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:40.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:40 smithi194 ceph-mon[32464]: osdmap e219: 8 total, 8 up, 8 in 2024-09-10T19:06:40.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi119-79477-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:40.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:40 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi119-79477-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:40.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:40 smithi119 ceph-mon[29577]: pgmap v291: 228 pgs: 32 creating+peering, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:06:40.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:40 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-46"}]': finished 2024-09-10T19:06:40.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:40 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi119-79477-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:40.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:40 smithi119 ceph-mon[29577]: osdmap e219: 8 total, 8 up, 8 in 2024-09-10T19:06:40.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi119-79477-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:40.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:40 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi119-79477-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:40.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:40 smithi119 ceph-mon[40674]: pgmap v291: 228 pgs: 32 creating+peering, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:06:40.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:40 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-46"}]': finished 2024-09-10T19:06:40.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:40 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi119-79477-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:40.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:40 smithi119 ceph-mon[40674]: osdmap e219: 8 total, 8 up, 8 in 2024-09-10T19:06:40.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi119-79477-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:40.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:40 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi119-79477-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:41.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:41 smithi119 ceph-mon[29577]: osdmap e220: 8 total, 8 up, 8 in 2024-09-10T19:06:41.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:41.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:41 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:41.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:41 smithi119 ceph-mon[40674]: osdmap e220: 8 total, 8 up, 8 in 2024-09-10T19:06:41.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:41.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:41 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:41.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:41 smithi194 ceph-mon[32464]: osdmap e220: 8 total, 8 up, 8 in 2024-09-10T19:06:41.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:41.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:41 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: pgmap v294: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi119-79477-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi119-79477-37"}]': finished 2024-09-10T19:06:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: osdmap e221: 8 total, 8 up, 8 in 2024-09-10T19:06:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-29"}]: dispatch 2024-09-10T19:06:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-29"}]: dispatch 2024-09-10T19:06:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-29"}]': finished 2024-09-10T19:06:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]': finished 2024-09-10T19:06:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: osdmap e222: 8 total, 8 up, 8 in 2024-09-10T19:06:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-29", "mode": "writeback"}]: dispatch 2024-09-10T19:06:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-29", "mode": "writeback"}]: dispatch 2024-09-10T19:06:42.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:42.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: pgmap v294: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi119-79477-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi119-79477-37"}]': finished 2024-09-10T19:06:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: osdmap e221: 8 total, 8 up, 8 in 2024-09-10T19:06:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-29"}]: dispatch 2024-09-10T19:06:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-29"}]: dispatch 2024-09-10T19:06:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-29"}]': finished 2024-09-10T19:06:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]': finished 2024-09-10T19:06:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: osdmap e222: 8 total, 8 up, 8 in 2024-09-10T19:06:42.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-29", "mode": "writeback"}]: dispatch 2024-09-10T19:06:42.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:42.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-29", "mode": "writeback"}]: dispatch 2024-09-10T19:06:42.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:42.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: pgmap v294: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 583 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi119-79477-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi119-79477-37"}]': finished 2024-09-10T19:06:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: osdmap e221: 8 total, 8 up, 8 in 2024-09-10T19:06:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-29"}]: dispatch 2024-09-10T19:06:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-29"}]: dispatch 2024-09-10T19:06:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-29"}]': finished 2024-09-10T19:06:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-46"}]': finished 2024-09-10T19:06:42.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: osdmap e222: 8 total, 8 up, 8 in 2024-09-10T19:06:42.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-29", "mode": "writeback"}]: dispatch 2024-09-10T19:06:42.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1207270993' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:42.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-29", "mode": "writeback"}]: dispatch 2024-09-10T19:06:42.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]: dispatch 2024-09-10T19:06:43.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:06:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:06:42] "GET /metrics HTTP/1.1" 200 230751 "" "Prometheus/2.43.0" 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-29", "mode": "writeback"}]': finished 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]': finished 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: osdmap e223: 8 total, 8 up, 8 in 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:43.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:43.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:43.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:43.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:43.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:43.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:06:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-29", "mode": "writeback"}]': finished 2024-09-10T19:06:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]': finished 2024-09-10T19:06:43.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: osdmap e223: 8 total, 8 up, 8 in 2024-09-10T19:06:43.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:43.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:43.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:43.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:43.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:43.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:43.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:43.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:43.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:43.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:06:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-29", "mode": "writeback"}]': finished 2024-09-10T19:06:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-46"}]': finished 2024-09-10T19:06:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: osdmap e223: 8 total, 8 up, 8 in 2024-09-10T19:06:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:43.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:43.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:43.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:43.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:44.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:44 smithi194 ceph-mon[32464]: pgmap v297: 236 pgs: 236 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 2 op/s 2024-09-10T19:06:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]': finished 2024-09-10T19:06:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:44 smithi194 ceph-mon[32464]: osdmap e224: 8 total, 8 up, 8 in 2024-09-10T19:06:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[40674]: pgmap v297: 236 pgs: 236 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 2 op/s 2024-09-10T19:06:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]': finished 2024-09-10T19:06:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[40674]: osdmap e224: 8 total, 8 up, 8 in 2024-09-10T19:06:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[29577]: pgmap v297: 236 pgs: 236 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.0 MiB/s wr, 2 op/s 2024-09-10T19:06:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi119-79477-37"}]': finished 2024-09-10T19:06:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi119-79499-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[29577]: osdmap e224: 8 total, 8 up, 8 in 2024-09-10T19:06:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2013549947' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]: dispatch 2024-09-10T19:06:44.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:46.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:46 smithi194 ceph-mon[32464]: pgmap v300: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-09-10T19:06:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]': finished 2024-09-10T19:06:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:46 smithi194 ceph-mon[32464]: osdmap e225: 8 total, 8 up, 8 in 2024-09-10T19:06:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi119-79477-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:46.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi119-79477-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[29577]: pgmap v300: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-09-10T19:06:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]': finished 2024-09-10T19:06:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[29577]: osdmap e225: 8 total, 8 up, 8 in 2024-09-10T19:06:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi119-79477-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi119-79477-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[40674]: pgmap v300: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-09-10T19:06:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi119-79477-37"}]': finished 2024-09-10T19:06:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[40674]: osdmap e225: 8 total, 8 up, 8 in 2024-09-10T19:06:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi119-79477-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi119-79477-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:47.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:47 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-47"}]': finished 2024-09-10T19:06:47.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:47 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi119-79477-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:47.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:47 smithi194 ceph-mon[32464]: osdmap e226: 8 total, 8 up, 8 in 2024-09-10T19:06:47.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi119-79477-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:47.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:47 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi119-79477-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:47.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:47.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:47 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:47.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:06:47.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:47.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:47 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-47"}]': finished 2024-09-10T19:06:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi119-79477-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[29577]: osdmap e226: 8 total, 8 up, 8 in 2024-09-10T19:06:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi119-79477-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi119-79477-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:06:47.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:47.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:47.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi119-79499-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi119-79499-47"}]': finished 2024-09-10T19:06:47.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi119-79477-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:47.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[40674]: osdmap e226: 8 total, 8 up, 8 in 2024-09-10T19:06:47.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi119-79477-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:47.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi119-79477-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:47.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:47.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:47.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:06:47.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:06:47.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:47 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:48.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:48 smithi119 ceph-mon[29577]: pgmap v303: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:48.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:48.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:48 smithi119 ceph-mon[29577]: osdmap e227: 8 total, 8 up, 8 in 2024-09-10T19:06:48.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29"}]: dispatch 2024-09-10T19:06:48.468 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29"}]: dispatch 2024-09-10T19:06:48.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:48 smithi119 ceph-mon[40674]: pgmap v303: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:48.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:48.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:48 smithi119 ceph-mon[40674]: osdmap e227: 8 total, 8 up, 8 in 2024-09-10T19:06:48.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29"}]: dispatch 2024-09-10T19:06:48.469 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29"}]: dispatch 2024-09-10T19:06:48.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:48 smithi194 ceph-mon[32464]: pgmap v303: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:48.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:48 smithi194 ceph-mon[32464]: osdmap e227: 8 total, 8 up, 8 in 2024-09-10T19:06:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29"}]: dispatch 2024-09-10T19:06:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29"}]: dispatch 2024-09-10T19:06:48.519 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:06:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:06:49.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:49 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi119-79477-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi119-79477-38"}]': finished 2024-09-10T19:06:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29"}]': finished 2024-09-10T19:06:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:49 smithi194 ceph-mon[32464]: osdmap e228: 8 total, 8 up, 8 in 2024-09-10T19:06:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:49.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:49 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:49.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:49 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi119-79477-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi119-79477-38"}]': finished 2024-09-10T19:06:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29"}]': finished 2024-09-10T19:06:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:49 smithi119 ceph-mon[29577]: osdmap e228: 8 total, 8 up, 8 in 2024-09-10T19:06:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:49 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:49.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:49 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:49.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi119-79477-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi119-79477-38"}]': finished 2024-09-10T19:06:49.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-29"}]': finished 2024-09-10T19:06:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:49 smithi119 ceph-mon[40674]: osdmap e228: 8 total, 8 up, 8 in 2024-09-10T19:06:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:49 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:50.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:50 smithi194 ceph-mon[32464]: pgmap v306: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-09-10T19:06:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:50 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]': finished 2024-09-10T19:06:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:50 smithi194 ceph-mon[32464]: osdmap e229: 8 total, 8 up, 8 in 2024-09-10T19:06:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:50 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:50.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:06:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:06:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:06:50.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:50 smithi119 ceph-mon[29577]: pgmap v306: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-09-10T19:06:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:50 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]': finished 2024-09-10T19:06:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:50 smithi119 ceph-mon[29577]: osdmap e229: 8 total, 8 up, 8 in 2024-09-10T19:06:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:50 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:50.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:50 smithi119 ceph-mon[40674]: pgmap v306: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-09-10T19:06:50.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:50 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi119-79499-47"}]': finished 2024-09-10T19:06:50.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:50 smithi119 ceph-mon[40674]: osdmap e229: 8 total, 8 up, 8 in 2024-09-10T19:06:50.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3807683869' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:50.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:50 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]: dispatch 2024-09-10T19:06:51.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]': finished 2024-09-10T19:06:51.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:51 smithi194 ceph-mon[32464]: osdmap e230: 8 total, 8 up, 8 in 2024-09-10T19:06:51.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:51.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:51.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:51.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:51.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:51.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:51.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:51.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:51.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi119-79499-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:51.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi119-79499-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:51.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]': finished 2024-09-10T19:06:51.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[29577]: osdmap e230: 8 total, 8 up, 8 in 2024-09-10T19:06:51.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:51.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:51.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:51.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:51.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:51.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:51.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:51.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:51.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi119-79499-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:51.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi119-79499-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:51.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi119-79499-47"}]': finished 2024-09-10T19:06:51.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[40674]: osdmap e230: 8 total, 8 up, 8 in 2024-09-10T19:06:51.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:51.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:51.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:51.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:51.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:51.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:51.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:51.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:51.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi119-79499-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:51.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi119-79499-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[29577]: pgmap v309: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-09-10T19:06:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]': finished 2024-09-10T19:06:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi119-79499-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[29577]: osdmap e231: 8 total, 8 up, 8 in 2024-09-10T19:06:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi119-79499-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi119-79499-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:52.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:52.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[40674]: pgmap v309: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-09-10T19:06:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]': finished 2024-09-10T19:06:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi119-79499-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[40674]: osdmap e231: 8 total, 8 up, 8 in 2024-09-10T19:06:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi119-79499-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi119-79499-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:52.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:52.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:52.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:52.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:52.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:52 smithi194 ceph-mon[32464]: pgmap v309: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-09-10T19:06:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:06:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi119-79477-38"}]': finished 2024-09-10T19:06:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi119-79499-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:52 smithi194 ceph-mon[32464]: osdmap e231: 8 total, 8 up, 8 in 2024-09-10T19:06:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi119-79499-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi119-79499-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3740275817' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]: dispatch 2024-09-10T19:06:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:06:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:52 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:06:53.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:06:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:06:52] "GET /metrics HTTP/1.1" 200 230780 "" "Prometheus/2.43.0" 2024-09-10T19:06:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]': finished 2024-09-10T19:06:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: osdmap e232: 8 total, 8 up, 8 in 2024-09-10T19:06:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-31"}]: dispatch 2024-09-10T19:06:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-31"}]: dispatch 2024-09-10T19:06:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi119-79477-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:53.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi119-79477-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:53.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi119-79499-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi119-79499-48"}]': finished 2024-09-10T19:06:53.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-31"}]': finished 2024-09-10T19:06:53.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi119-79477-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:53.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: osdmap e233: 8 total, 8 up, 8 in 2024-09-10T19:06:53.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-31", "mode": "writeback"}]: dispatch 2024-09-10T19:06:53.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-31", "mode": "writeback"}]: dispatch 2024-09-10T19:06:53.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi119-79477-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi119-79477-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]': finished 2024-09-10T19:06:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: osdmap e232: 8 total, 8 up, 8 in 2024-09-10T19:06:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-31"}]: dispatch 2024-09-10T19:06:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-31"}]: dispatch 2024-09-10T19:06:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi119-79477-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:53.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi119-79477-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:53.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi119-79499-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi119-79499-48"}]': finished 2024-09-10T19:06:53.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-31"}]': finished 2024-09-10T19:06:53.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi119-79477-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:53.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: osdmap e233: 8 total, 8 up, 8 in 2024-09-10T19:06:53.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-31", "mode": "writeback"}]: dispatch 2024-09-10T19:06:53.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-31", "mode": "writeback"}]: dispatch 2024-09-10T19:06:53.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi119-79477-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi119-79477-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi119-79477-38"}]': finished 2024-09-10T19:06:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:06:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: osdmap e232: 8 total, 8 up, 8 in 2024-09-10T19:06:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-31"}]: dispatch 2024-09-10T19:06:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-31"}]: dispatch 2024-09-10T19:06:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi119-79477-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:53.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi119-79477-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:53.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi119-79499-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi119-79499-48"}]': finished 2024-09-10T19:06:53.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-31"}]': finished 2024-09-10T19:06:53.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi119-79477-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:53.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: osdmap e233: 8 total, 8 up, 8 in 2024-09-10T19:06:53.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-31", "mode": "writeback"}]: dispatch 2024-09-10T19:06:53.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-31", "mode": "writeback"}]: dispatch 2024-09-10T19:06:53.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi119-79477-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:53.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi119-79477-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:54.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:54 smithi194 ceph-mon[32464]: pgmap v312: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:54.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:54 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:54 smithi119 ceph-mon[29577]: pgmap v312: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:54 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:54 smithi119 ceph-mon[40674]: pgmap v312: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:54 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:06:55.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:55 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-31", "mode": "writeback"}]': finished 2024-09-10T19:06:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:55 smithi194 ceph-mon[32464]: osdmap e234: 8 total, 8 up, 8 in 2024-09-10T19:06:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:55 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:55 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi119-79477-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi119-79477-39"}]': finished 2024-09-10T19:06:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:55 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:55 smithi194 ceph-mon[32464]: osdmap e235: 8 total, 8 up, 8 in 2024-09-10T19:06:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31"}]: dispatch 2024-09-10T19:06:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:55 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:55 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31"}]: dispatch 2024-09-10T19:06:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-31", "mode": "writeback"}]': finished 2024-09-10T19:06:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[29577]: osdmap e234: 8 total, 8 up, 8 in 2024-09-10T19:06:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi119-79477-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi119-79477-39"}]': finished 2024-09-10T19:06:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[29577]: osdmap e235: 8 total, 8 up, 8 in 2024-09-10T19:06:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31"}]: dispatch 2024-09-10T19:06:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31"}]: dispatch 2024-09-10T19:06:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-31", "mode": "writeback"}]': finished 2024-09-10T19:06:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[40674]: osdmap e234: 8 total, 8 up, 8 in 2024-09-10T19:06:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:06:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi119-79477-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi119-79477-39"}]': finished 2024-09-10T19:06:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:06:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[40674]: osdmap e235: 8 total, 8 up, 8 in 2024-09-10T19:06:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31"}]: dispatch 2024-09-10T19:06:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:55.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:55 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31"}]: dispatch 2024-09-10T19:06:56.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:56 smithi194 ceph-mon[32464]: pgmap v315: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:56 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:56 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]': finished 2024-09-10T19:06:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:56 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31"}]': finished 2024-09-10T19:06:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:56 smithi194 ceph-mon[32464]: osdmap e236: 8 total, 8 up, 8 in 2024-09-10T19:06:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:56 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:56.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:56 smithi119 ceph-mon[29577]: pgmap v315: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:56 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:56 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]': finished 2024-09-10T19:06:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:56 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31"}]': finished 2024-09-10T19:06:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:56 smithi119 ceph-mon[29577]: osdmap e236: 8 total, 8 up, 8 in 2024-09-10T19:06:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:56 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:56 smithi119 ceph-mon[40674]: pgmap v315: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:06:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:56 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:06:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:56 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi119-79499-48"}]': finished 2024-09-10T19:06:56.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:56 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-31"}]': finished 2024-09-10T19:06:56.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:56 smithi119 ceph-mon[40674]: osdmap e236: 8 total, 8 up, 8 in 2024-09-10T19:06:56.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2145682929' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:56.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:56 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]: dispatch 2024-09-10T19:06:58.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:06:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:06:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:58 smithi194 ceph-mon[32464]: pgmap v318: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]': finished 2024-09-10T19:06:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:58 smithi194 ceph-mon[32464]: osdmap e237: 8 total, 8 up, 8 in 2024-09-10T19:06:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:58.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:58.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:58.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi119-79499-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:58.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi119-79499-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:58.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[29577]: pgmap v318: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:58.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]': finished 2024-09-10T19:06:58.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[29577]: osdmap e237: 8 total, 8 up, 8 in 2024-09-10T19:06:58.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:58.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:58.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:58.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:58.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:58.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:58.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi119-79499-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:58.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi119-79499-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:58.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[40674]: pgmap v318: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:06:58.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi119-79499-48"}]': finished 2024-09-10T19:06:58.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[40674]: osdmap e237: 8 total, 8 up, 8 in 2024-09-10T19:06:58.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:58.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:58.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:58.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:58.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:58.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:58.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi119-79499-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:58.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi119-79499-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:06:59.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]': finished 2024-09-10T19:06:59.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi119-79499-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:59.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:59 smithi194 ceph-mon[32464]: osdmap e238: 8 total, 8 up, 8 in 2024-09-10T19:06:59.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:59 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi119-79499-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:59.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:59 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:59.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:59 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:59.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi119-79499-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:59.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:59.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:59.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:06:59 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:59.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]': finished 2024-09-10T19:06:59.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi119-79499-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:59.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[29577]: osdmap e238: 8 total, 8 up, 8 in 2024-09-10T19:06:59.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi119-79499-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:59.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:59.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:59.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi119-79499-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:59.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:59.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:59.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:06:59.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi119-79477-39"}]': finished 2024-09-10T19:06:59.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi119-79499-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:06:59.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[40674]: osdmap e238: 8 total, 8 up, 8 in 2024-09-10T19:06:59.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi119-79499-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:59.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1411147189' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:59.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:59.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi119-79499-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:06:59.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]: dispatch 2024-09-10T19:06:59.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:06:59.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:06:59 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:00.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:07:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:07:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:07:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:00 smithi194 ceph-mon[32464]: pgmap v321: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:07:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:00 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]': finished 2024-09-10T19:07:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:00 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:00.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:00 smithi194 ceph-mon[32464]: osdmap e239: 8 total, 8 up, 8 in 2024-09-10T19:07:00.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:00.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:00 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:00.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:00.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:00.521 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi119-79477-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[29577]: pgmap v321: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:07:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]': finished 2024-09-10T19:07:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[29577]: osdmap e239: 8 total, 8 up, 8 in 2024-09-10T19:07:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi119-79477-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[40674]: pgmap v321: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:07:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi119-79477-39"}]': finished 2024-09-10T19:07:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[40674]: osdmap e239: 8 total, 8 up, 8 in 2024-09-10T19:07:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi119-79477-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:01.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:01 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi119-79499-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi119-79499-49"}]': finished 2024-09-10T19:07:01.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:01 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:01.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi119-79477-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:01.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:01 smithi194 ceph-mon[32464]: osdmap e240: 8 total, 8 up, 8 in 2024-09-10T19:07:01.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi119-79477-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:01.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-33"}]: dispatch 2024-09-10T19:07:01.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:01 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-33"}]: dispatch 2024-09-10T19:07:01.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:01 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi119-79499-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi119-79499-49"}]': finished 2024-09-10T19:07:01.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:01 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:01.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi119-79477-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:01.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:01 smithi119 ceph-mon[29577]: osdmap e240: 8 total, 8 up, 8 in 2024-09-10T19:07:01.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi119-79477-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:01.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-33"}]: dispatch 2024-09-10T19:07:01.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:01 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-33"}]: dispatch 2024-09-10T19:07:01.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:01 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi119-79499-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi119-79499-49"}]': finished 2024-09-10T19:07:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:01 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi119-79477-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:01 smithi119 ceph-mon[40674]: osdmap e240: 8 total, 8 up, 8 in 2024-09-10T19:07:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi119-79477-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-33"}]: dispatch 2024-09-10T19:07:01.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:01 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-33"}]: dispatch 2024-09-10T19:07:02.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:02 smithi194 ceph-mon[32464]: pgmap v324: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:07:02.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:02 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-33"}]': finished 2024-09-10T19:07:02.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:02 smithi194 ceph-mon[32464]: osdmap e241: 8 total, 8 up, 8 in 2024-09-10T19:07:02.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-33", "mode": "writeback"}]: dispatch 2024-09-10T19:07:02.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:02 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-33", "mode": "writeback"}]: dispatch 2024-09-10T19:07:02.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:02.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-mon[29577]: pgmap v324: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:07:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-33"}]': finished 2024-09-10T19:07:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-mon[29577]: osdmap e241: 8 total, 8 up, 8 in 2024-09-10T19:07:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-33", "mode": "writeback"}]: dispatch 2024-09-10T19:07:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-33", "mode": "writeback"}]: dispatch 2024-09-10T19:07:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:02.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-mon[40674]: pgmap v324: 236 pgs: 40 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:07:02.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-33"}]': finished 2024-09-10T19:07:02.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-mon[40674]: osdmap e241: 8 total, 8 up, 8 in 2024-09-10T19:07:02.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-33", "mode": "writeback"}]: dispatch 2024-09-10T19:07:02.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-33", "mode": "writeback"}]: dispatch 2024-09-10T19:07:02.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:02.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:03.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:07:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:07:02] "GET /metrics HTTP/1.1" 200 230781 "" "Prometheus/2.43.0" 2024-09-10T19:07:03.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:03 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:03.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi119-79477-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi119-79477-40"}]': finished 2024-09-10T19:07:03.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-33", "mode": "writeback"}]': finished 2024-09-10T19:07:03.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:03 smithi194 ceph-mon[32464]: osdmap e242: 8 total, 8 up, 8 in 2024-09-10T19:07:03.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:07:03.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:07:03.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:03.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi119-79477-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi119-79477-40"}]': finished 2024-09-10T19:07:03.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-33", "mode": "writeback"}]': finished 2024-09-10T19:07:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[29577]: osdmap e242: 8 total, 8 up, 8 in 2024-09-10T19:07:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:07:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:07:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:03.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:03.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi119-79477-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi119-79477-40"}]': finished 2024-09-10T19:07:03.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-33", "mode": "writeback"}]': finished 2024-09-10T19:07:03.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[40674]: osdmap e242: 8 total, 8 up, 8 in 2024-09-10T19:07:03.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:07:03.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:07:03.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:03.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:04.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:04 smithi194 ceph-mon[32464]: pgmap v327: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-10T19:07:04.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]': finished 2024-09-10T19:07:04.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:04.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:04 smithi194 ceph-mon[32464]: osdmap e243: 8 total, 8 up, 8 in 2024-09-10T19:07:04.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33"}]: dispatch 2024-09-10T19:07:04.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33"}]: dispatch 2024-09-10T19:07:04.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:07:04.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:07:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[29577]: pgmap v327: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-10T19:07:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]': finished 2024-09-10T19:07:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[29577]: osdmap e243: 8 total, 8 up, 8 in 2024-09-10T19:07:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33"}]: dispatch 2024-09-10T19:07:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33"}]: dispatch 2024-09-10T19:07:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:07:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:07:04.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[40674]: pgmap v327: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-10T19:07:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi119-79499-49"}]': finished 2024-09-10T19:07:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[40674]: osdmap e243: 8 total, 8 up, 8 in 2024-09-10T19:07:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33"}]: dispatch 2024-09-10T19:07:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33"}]: dispatch 2024-09-10T19:07:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/133835027' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:07:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]: dispatch 2024-09-10T19:07:05.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:05 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33"}]': finished 2024-09-10T19:07:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]': finished 2024-09-10T19:07:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:05 smithi194 ceph-mon[32464]: osdmap e244: 8 total, 8 up, 8 in 2024-09-10T19:07:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi119-79499-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:05.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi119-79499-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33"}]': finished 2024-09-10T19:07:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]': finished 2024-09-10T19:07:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[29577]: osdmap e244: 8 total, 8 up, 8 in 2024-09-10T19:07:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi119-79499-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:05.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi119-79499-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-33"}]': finished 2024-09-10T19:07:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi119-79499-49"}]': finished 2024-09-10T19:07:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[40674]: osdmap e244: 8 total, 8 up, 8 in 2024-09-10T19:07:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:05.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:05.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:05.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi119-79499-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:05.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi119-79499-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:06.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:06 smithi194 ceph-mon[32464]: pgmap v330: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-10T19:07:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:06 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi119-79477-40"}]': finished 2024-09-10T19:07:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:06 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi119-79499-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:06 smithi194 ceph-mon[32464]: osdmap e245: 8 total, 8 up, 8 in 2024-09-10T19:07:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi119-79499-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:06 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi119-79499-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[29577]: pgmap v330: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-10T19:07:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi119-79477-40"}]': finished 2024-09-10T19:07:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi119-79499-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[29577]: osdmap e245: 8 total, 8 up, 8 in 2024-09-10T19:07:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi119-79499-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi119-79499-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[40674]: pgmap v330: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 4 op/s 2024-09-10T19:07:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi119-79477-40"}]': finished 2024-09-10T19:07:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi119-79499-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[40674]: osdmap e245: 8 total, 8 up, 8 in 2024-09-10T19:07:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi119-79499-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi119-79477-40"}]: dispatch 2024-09-10T19:07:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:06 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi119-79499-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:07.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi119-79477-40"}]': finished 2024-09-10T19:07:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:07 smithi194 ceph-mon[32464]: osdmap e246: 8 total, 8 up, 8 in 2024-09-10T19:07:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi119-79477-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi119-79499-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi119-79499-50"}]': finished 2024-09-10T19:07:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi119-79477-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:07 smithi194 ceph-mon[32464]: osdmap e247: 8 total, 8 up, 8 in 2024-09-10T19:07:07.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi119-79477-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:07.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:07.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi119-79477-40"}]': finished 2024-09-10T19:07:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[29577]: osdmap e246: 8 total, 8 up, 8 in 2024-09-10T19:07:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi119-79477-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi119-79499-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi119-79499-50"}]': finished 2024-09-10T19:07:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi119-79477-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:07.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[29577]: osdmap e247: 8 total, 8 up, 8 in 2024-09-10T19:07:07.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi119-79477-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:07.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:07.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:07.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2838928832' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi119-79477-40"}]': finished 2024-09-10T19:07:07.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[40674]: osdmap e246: 8 total, 8 up, 8 in 2024-09-10T19:07:07.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:07.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:07.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:07.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:07.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi119-79477-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:07.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi119-79499-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi119-79499-50"}]': finished 2024-09-10T19:07:07.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:07.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi119-79477-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:07.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[40674]: osdmap e247: 8 total, 8 up, 8 in 2024-09-10T19:07:07.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi119-79477-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:07.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:07.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:08.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:07:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:07:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:08 smithi194 ceph-mon[32464]: pgmap v333: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:08 smithi194 ceph-mon[32464]: osdmap e248: 8 total, 8 up, 8 in 2024-09-10T19:07:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:08 smithi119 ceph-mon[29577]: pgmap v333: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:08 smithi119 ceph-mon[29577]: osdmap e248: 8 total, 8 up, 8 in 2024-09-10T19:07:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:08 smithi119 ceph-mon[40674]: pgmap v333: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:08 smithi119 ceph-mon[40674]: osdmap e248: 8 total, 8 up, 8 in 2024-09-10T19:07:08.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:08.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi119-79477-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi119-79477-41"}]': finished 2024-09-10T19:07:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]': finished 2024-09-10T19:07:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[29577]: osdmap e249: 8 total, 8 up, 8 in 2024-09-10T19:07:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-35", "mode": "writeback"}]: dispatch 2024-09-10T19:07:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-35", "mode": "writeback"}]: dispatch 2024-09-10T19:07:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi119-79477-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi119-79477-41"}]': finished 2024-09-10T19:07:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]': finished 2024-09-10T19:07:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[40674]: osdmap e249: 8 total, 8 up, 8 in 2024-09-10T19:07:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-35", "mode": "writeback"}]: dispatch 2024-09-10T19:07:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-35", "mode": "writeback"}]: dispatch 2024-09-10T19:07:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:09 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi119-79477-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi119-79477-41"}]': finished 2024-09-10T19:07:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]': finished 2024-09-10T19:07:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:09 smithi194 ceph-mon[32464]: osdmap e249: 8 total, 8 up, 8 in 2024-09-10T19:07:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-35", "mode": "writeback"}]: dispatch 2024-09-10T19:07:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-35", "mode": "writeback"}]: dispatch 2024-09-10T19:07:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:10.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:07:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:07:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:07:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:10 smithi119 ceph-mon[29577]: pgmap v336: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:07:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:10 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-35", "mode": "writeback"}]': finished 2024-09-10T19:07:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]': finished 2024-09-10T19:07:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:10 smithi119 ceph-mon[29577]: osdmap e250: 8 total, 8 up, 8 in 2024-09-10T19:07:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:10.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:10.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:10 smithi119 ceph-mon[40674]: pgmap v336: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:07:10.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:10 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:10.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-35", "mode": "writeback"}]': finished 2024-09-10T19:07:10.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]': finished 2024-09-10T19:07:10.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:10 smithi119 ceph-mon[40674]: osdmap e250: 8 total, 8 up, 8 in 2024-09-10T19:07:10.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:10.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:11.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:10 smithi194 ceph-mon[32464]: pgmap v336: 236 pgs: 8 creating+peering, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:07:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:10 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-35", "mode": "writeback"}]': finished 2024-09-10T19:07:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi119-79499-50"}]': finished 2024-09-10T19:07:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:10 smithi194 ceph-mon[32464]: osdmap e250: 8 total, 8 up, 8 in 2024-09-10T19:07:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/848597110' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]: dispatch 2024-09-10T19:07:12.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:12 smithi119 ceph-mon[29577]: pgmap v339: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:12.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]': finished 2024-09-10T19:07:12.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:12 smithi119 ceph-mon[29577]: osdmap e251: 8 total, 8 up, 8 in 2024-09-10T19:07:12.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:12.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:12 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:12.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:12 smithi119 ceph-mon[40674]: pgmap v339: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:12.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]': finished 2024-09-10T19:07:12.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:12 smithi119 ceph-mon[40674]: osdmap e251: 8 total, 8 up, 8 in 2024-09-10T19:07:12.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:12.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:12 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:12.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:12 smithi194 ceph-mon[32464]: pgmap v339: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi119-79499-50"}]': finished 2024-09-10T19:07:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:12 smithi194 ceph-mon[32464]: osdmap e251: 8 total, 8 up, 8 in 2024-09-10T19:07:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:12 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:13.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:07:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:07:12] "GET /metrics HTTP/1.1" 200 230781 "" "Prometheus/2.43.0" 2024-09-10T19:07:13.298 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAiote 2024-09-10T19:07:13.298 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6981 ms) 2024-09-10T19:07:13.299 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-10T19:07:13.299 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7020 ms) 2024-09-10T19:07:13.299 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-10T19:07:13.299 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6982 ms) 2024-09-10T19:07:13.299 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-10T19:07:13.299 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7052 ms) 2024-09-10T19:07:13.299 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-10T19:07:13.299 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7041 ms) 2024-09-10T19:07:13.299 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-10T19:07:13.299 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7019 ms) 2024-09-10T19:07:13.300 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-10T19:07:13.300 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7026 ms) 2024-09-10T19:07:13.300 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-10T19:07:13.300 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6874 ms) 2024-09-10T19:07:13.300 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-10T19:07:13.300 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7037 ms) 2024-09-10T19:07:13.300 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-10T19:07:13.300 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7058 ms) 2024-09-10T19:07:13.300 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-10T19:07:13.300 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6908 ms) 2024-09-10T19:07:13.300 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-10T19:07:13.300 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7237 ms) 2024-09-10T19:07:13.300 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-10T19:07:13.301 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7019 ms) 2024-09-10T19:07:13.301 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-10T19:07:13.301 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7056 ms) 2024-09-10T19:07:13.301 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-10T19:07:13.301 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6780 ms) 2024-09-10T19:07:13.301 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112103 ms total) 2024-09-10T19:07:13.301 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: 2024-09-10T19:07:13.301 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [----------] Global test environment tear-down 2024-09-10T19:07:13.301 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [==========] 41 tests from 2 test suites ran. (192048 ms total) 2024-09-10T19:07:13.301 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio: [ PASSED ] 41 tests. 2024-09-10T19:07:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi119-79477-41"}]': finished 2024-09-10T19:07:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:13 smithi119 ceph-mon[29577]: osdmap e252: 8 total, 8 up, 8 in 2024-09-10T19:07:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3565983307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:13.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:13.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi119-79477-41"}]': finished 2024-09-10T19:07:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:13 smithi119 ceph-mon[40674]: osdmap e252: 8 total, 8 up, 8 in 2024-09-10T19:07:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3565983307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:13.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:13.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi119-79477-41"}]': finished 2024-09-10T19:07:13.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:13 smithi194 ceph-mon[32464]: osdmap e252: 8 total, 8 up, 8 in 2024-09-10T19:07:13.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi119-79477-41"}]: dispatch 2024-09-10T19:07:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3565983307' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:14 smithi119 ceph-mon[29577]: pgmap v342: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:07:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi119-79477-41"}]': finished 2024-09-10T19:07:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:14 smithi119 ceph-mon[29577]: osdmap e253: 8 total, 8 up, 8 in 2024-09-10T19:07:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:14.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:14.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:14 smithi119 ceph-mon[40674]: pgmap v342: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:07:14.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi119-79477-41"}]': finished 2024-09-10T19:07:14.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:14.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:14.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:14 smithi119 ceph-mon[40674]: osdmap e253: 8 total, 8 up, 8 in 2024-09-10T19:07:14.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:14.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:14.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:14 smithi194 ceph-mon[32464]: pgmap v342: 260 pgs: 32 creating+peering, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:07:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3628226994' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi119-79477-41"}]': finished 2024-09-10T19:07:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi119-79499-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:14 smithi194 ceph-mon[32464]: osdmap e253: 8 total, 8 up, 8 in 2024-09-10T19:07:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]': finished 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[29577]: osdmap e254: 8 total, 8 up, 8 in 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:15.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:15.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:15.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[29577]: osdmap e255: 8 total, 8 up, 8 in 2024-09-10T19:07:15.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi119-79499-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi119-79499-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]': finished 2024-09-10T19:07:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[40674]: osdmap e254: 8 total, 8 up, 8 in 2024-09-10T19:07:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:15.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:15.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:15.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[40674]: osdmap e255: 8 total, 8 up, 8 in 2024-09-10T19:07:15.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi119-79499-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi119-79499-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-35"}]': finished 2024-09-10T19:07:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:15 smithi194 ceph-mon[32464]: osdmap e254: 8 total, 8 up, 8 in 2024-09-10T19:07:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:15 smithi194 ceph-mon[32464]: osdmap e255: 8 total, 8 up, 8 in 2024-09-10T19:07:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi119-79499-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi119-79499-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:16.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:16 smithi119 ceph-mon[29577]: pgmap v345: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:07:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:16 smithi119 ceph-mon[40674]: pgmap v345: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:07:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:17.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:16 smithi194 ceph-mon[32464]: pgmap v345: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:07:17.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:17.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[29577]: osdmap e256: 8 total, 8 up, 8 in 2024-09-10T19:07:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi119-79499-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]': finished 2024-09-10T19:07:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35"}]': finished 2024-09-10T19:07:17.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[29577]: osdmap e257: 8 total, 8 up, 8 in 2024-09-10T19:07:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[40674]: osdmap e256: 8 total, 8 up, 8 in 2024-09-10T19:07:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi119-79499-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]': finished 2024-09-10T19:07:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35"}]': finished 2024-09-10T19:07:17.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:17 smithi119 ceph-mon[40674]: osdmap e257: 8 total, 8 up, 8 in 2024-09-10T19:07:18.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:17 smithi194 ceph-mon[32464]: osdmap e256: 8 total, 8 up, 8 in 2024-09-10T19:07:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35"}]: dispatch 2024-09-10T19:07:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:17 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:17 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi119-79499-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]': finished 2024-09-10T19:07:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-35"}]': finished 2024-09-10T19:07:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:17 smithi194 ceph-mon[32464]: osdmap e257: 8 total, 8 up, 8 in 2024-09-10T19:07:18.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:07:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:07:18.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:18 smithi194 ceph-mon[32464]: pgmap v348: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:18.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:18 smithi194 ceph-mon[32464]: osdmap e258: 8 total, 8 up, 8 in 2024-09-10T19:07:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:18 smithi119 ceph-mon[29577]: pgmap v348: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:18 smithi119 ceph-mon[29577]: osdmap e258: 8 total, 8 up, 8 in 2024-09-10T19:07:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:18 smithi119 ceph-mon[40674]: pgmap v348: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:18 smithi119 ceph-mon[40674]: osdmap e258: 8 total, 8 up, 8 in 2024-09-10T19:07:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:19 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:19 smithi119 ceph-mon[29577]: osdmap e259: 8 total, 8 up, 8 in 2024-09-10T19:07:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:19.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:19 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:19 smithi119 ceph-mon[40674]: osdmap e259: 8 total, 8 up, 8 in 2024-09-10T19:07:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:20.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:19 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:20.115 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:19 smithi194 ceph-mon[32464]: osdmap e259: 8 total, 8 up, 8 in 2024-09-10T19:07:20.116 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:20.116 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:20.116 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:20.116 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:20.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:07:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:07:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:07:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[29577]: pgmap v351: 204 pgs: 8 creating+peering, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]': finished 2024-09-10T19:07:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[29577]: osdmap e260: 8 total, 8 up, 8 in 2024-09-10T19:07:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[40674]: pgmap v351: 204 pgs: 8 creating+peering, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]': finished 2024-09-10T19:07:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[40674]: osdmap e260: 8 total, 8 up, 8 in 2024-09-10T19:07:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:21.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:20 smithi194 ceph-mon[32464]: pgmap v351: 204 pgs: 8 creating+peering, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi119-79499-52"}]': finished 2024-09-10T19:07:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:20 smithi194 ceph-mon[32464]: osdmap e260: 8 total, 8 up, 8 in 2024-09-10T19:07:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2043806922' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]: dispatch 2024-09-10T19:07:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: pgmap v354: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]': finished 2024-09-10T19:07:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: osdmap e261: 8 total, 8 up, 8 in 2024-09-10T19:07:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-37"}]: dispatch 2024-09-10T19:07:22.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-37"}]: dispatch 2024-09-10T19:07:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-37"}]': finished 2024-09-10T19:07:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: osdmap e262: 8 total, 8 up, 8 in 2024-09-10T19:07:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-37", "mode": "writeback"}]: dispatch 2024-09-10T19:07:22.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-37", "mode": "writeback"}]: dispatch 2024-09-10T19:07:22.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi119-79499-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi119-79499-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: pgmap v354: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]': finished 2024-09-10T19:07:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: osdmap e261: 8 total, 8 up, 8 in 2024-09-10T19:07:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-37"}]: dispatch 2024-09-10T19:07:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-37"}]: dispatch 2024-09-10T19:07:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:22.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:22.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-37"}]': finished 2024-09-10T19:07:22.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:22.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: osdmap e262: 8 total, 8 up, 8 in 2024-09-10T19:07:22.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-37", "mode": "writeback"}]: dispatch 2024-09-10T19:07:22.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-37", "mode": "writeback"}]: dispatch 2024-09-10T19:07:22.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi119-79499-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi119-79499-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: pgmap v354: 228 pgs: 32 unknown, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi119-79499-52"}]': finished 2024-09-10T19:07:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: osdmap e261: 8 total, 8 up, 8 in 2024-09-10T19:07:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-37"}]: dispatch 2024-09-10T19:07:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-37"}]: dispatch 2024-09-10T19:07:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:22.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:22.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-37"}]': finished 2024-09-10T19:07:22.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:22.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: osdmap e262: 8 total, 8 up, 8 in 2024-09-10T19:07:22.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-37", "mode": "writeback"}]: dispatch 2024-09-10T19:07:22.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-37", "mode": "writeback"}]: dispatch 2024-09-10T19:07:22.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi119-79499-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:22.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi119-79499-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:23.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:07:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:07:22] "GET /metrics HTTP/1.1" 200 228395 "" "Prometheus/2.43.0" 2024-09-10T19:07:23.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:23 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:23.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:23 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:23.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:23 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:24.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:24 smithi194 ceph-mon[32464]: pgmap v357: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-37", "mode": "writeback"}]': finished 2024-09-10T19:07:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:24 smithi194 ceph-mon[32464]: osdmap e263: 8 total, 8 up, 8 in 2024-09-10T19:07:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi119-79499-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi119-79499-53"}]': finished 2024-09-10T19:07:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:24 smithi194 ceph-mon[32464]: osdmap e264: 8 total, 8 up, 8 in 2024-09-10T19:07:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37"}]: dispatch 2024-09-10T19:07:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37"}]: dispatch 2024-09-10T19:07:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[29577]: pgmap v357: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-37", "mode": "writeback"}]': finished 2024-09-10T19:07:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[29577]: osdmap e263: 8 total, 8 up, 8 in 2024-09-10T19:07:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi119-79499-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi119-79499-53"}]': finished 2024-09-10T19:07:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[29577]: osdmap e264: 8 total, 8 up, 8 in 2024-09-10T19:07:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37"}]: dispatch 2024-09-10T19:07:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37"}]: dispatch 2024-09-10T19:07:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[40674]: pgmap v357: 228 pgs: 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-37", "mode": "writeback"}]': finished 2024-09-10T19:07:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[40674]: osdmap e263: 8 total, 8 up, 8 in 2024-09-10T19:07:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi119-79499-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi119-79499-53"}]': finished 2024-09-10T19:07:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[40674]: osdmap e264: 8 total, 8 up, 8 in 2024-09-10T19:07:24.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37"}]: dispatch 2024-09-10T19:07:24.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37"}]: dispatch 2024-09-10T19:07:25.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:25 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:25.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37"}]': finished 2024-09-10T19:07:25.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:25 smithi194 ceph-mon[32464]: osdmap e265: 8 total, 8 up, 8 in 2024-09-10T19:07:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:25 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37"}]': finished 2024-09-10T19:07:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:25 smithi119 ceph-mon[29577]: osdmap e265: 8 total, 8 up, 8 in 2024-09-10T19:07:25.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:25 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:25.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-37"}]': finished 2024-09-10T19:07:25.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:25 smithi119 ceph-mon[40674]: osdmap e265: 8 total, 8 up, 8 in 2024-09-10T19:07:26.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:26 smithi194 ceph-mon[32464]: pgmap v360: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:26 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:26 smithi194 ceph-mon[32464]: osdmap e266: 8 total, 8 up, 8 in 2024-09-10T19:07:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:26.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:26 smithi119 ceph-mon[29577]: pgmap v360: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:26 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:26 smithi119 ceph-mon[29577]: osdmap e266: 8 total, 8 up, 8 in 2024-09-10T19:07:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:26 smithi119 ceph-mon[40674]: pgmap v360: 236 pgs: 8 unknown, 228 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:26 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:26 smithi119 ceph-mon[40674]: osdmap e266: 8 total, 8 up, 8 in 2024-09-10T19:07:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:26.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:28.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:07:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:07:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:28 smithi194 ceph-mon[32464]: pgmap v363: 196 pgs: 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]': finished 2024-09-10T19:07:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:28 smithi194 ceph-mon[32464]: osdmap e267: 8 total, 8 up, 8 in 2024-09-10T19:07:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:28.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:28 smithi119 ceph-mon[29577]: pgmap v363: 196 pgs: 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]': finished 2024-09-10T19:07:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:28 smithi119 ceph-mon[29577]: osdmap e267: 8 total, 8 up, 8 in 2024-09-10T19:07:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:28.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:28.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:28 smithi119 ceph-mon[40674]: pgmap v363: 196 pgs: 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:28.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi119-79499-53"}]': finished 2024-09-10T19:07:28.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:28 smithi119 ceph-mon[40674]: osdmap e267: 8 total, 8 up, 8 in 2024-09-10T19:07:28.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:28.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/994815267' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:28.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:28.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]: dispatch 2024-09-10T19:07:29.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]': finished 2024-09-10T19:07:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: osdmap e268: 8 total, 8 up, 8 in 2024-09-10T19:07:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi119-79499-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:29.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi119-79499-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:29.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:29.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:29.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi119-79499-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:29.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: osdmap e269: 8 total, 8 up, 8 in 2024-09-10T19:07:29.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-39"}]: dispatch 2024-09-10T19:07:29.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-39"}]: dispatch 2024-09-10T19:07:29.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi119-79499-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi119-79499-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]': finished 2024-09-10T19:07:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: osdmap e268: 8 total, 8 up, 8 in 2024-09-10T19:07:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi119-79499-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi119-79499-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi119-79499-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: osdmap e269: 8 total, 8 up, 8 in 2024-09-10T19:07:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-39"}]: dispatch 2024-09-10T19:07:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-39"}]: dispatch 2024-09-10T19:07:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi119-79499-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi119-79499-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi119-79499-53"}]': finished 2024-09-10T19:07:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: osdmap e268: 8 total, 8 up, 8 in 2024-09-10T19:07:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi119-79499-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi119-79499-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:29.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:29.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi119-79499-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:29.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: osdmap e269: 8 total, 8 up, 8 in 2024-09-10T19:07:29.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-39"}]: dispatch 2024-09-10T19:07:29.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-39"}]: dispatch 2024-09-10T19:07:29.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi119-79499-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:29.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi119-79499-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:30.455 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:07:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:07:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:07:30.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:30 smithi194 ceph-mon[32464]: pgmap v366: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:07:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:30 smithi119 ceph-mon[29577]: pgmap v366: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:07:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:30 smithi119 ceph-mon[40674]: pgmap v366: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:07:31.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-39"}]': finished 2024-09-10T19:07:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-39", "mode": "writeback"}]: dispatch 2024-09-10T19:07:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:31 smithi194 ceph-mon[32464]: osdmap e270: 8 total, 8 up, 8 in 2024-09-10T19:07:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-39", "mode": "writeback"}]: dispatch 2024-09-10T19:07:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:31 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi119-79499-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi119-79499-54"}]': finished 2024-09-10T19:07:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-39", "mode": "writeback"}]': finished 2024-09-10T19:07:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:31 smithi194 ceph-mon[32464]: osdmap e271: 8 total, 8 up, 8 in 2024-09-10T19:07:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:31.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-39"}]': finished 2024-09-10T19:07:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-39", "mode": "writeback"}]: dispatch 2024-09-10T19:07:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[29577]: osdmap e270: 8 total, 8 up, 8 in 2024-09-10T19:07:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-39", "mode": "writeback"}]: dispatch 2024-09-10T19:07:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi119-79499-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi119-79499-54"}]': finished 2024-09-10T19:07:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-39", "mode": "writeback"}]': finished 2024-09-10T19:07:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[29577]: osdmap e271: 8 total, 8 up, 8 in 2024-09-10T19:07:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-39"}]': finished 2024-09-10T19:07:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-39", "mode": "writeback"}]: dispatch 2024-09-10T19:07:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[40674]: osdmap e270: 8 total, 8 up, 8 in 2024-09-10T19:07:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-39", "mode": "writeback"}]: dispatch 2024-09-10T19:07:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:07:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi119-79499-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi119-79499-54"}]': finished 2024-09-10T19:07:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-39", "mode": "writeback"}]': finished 2024-09-10T19:07:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[40674]: osdmap e271: 8 total, 8 up, 8 in 2024-09-10T19:07:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:32 smithi194 ceph-mon[32464]: pgmap v369: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:07:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:32 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:32 smithi194 ceph-mon[32464]: osdmap e272: 8 total, 8 up, 8 in 2024-09-10T19:07:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39"}]: dispatch 2024-09-10T19:07:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39"}]: dispatch 2024-09-10T19:07:32.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:07:32] "GET /metrics HTTP/1.1" 200 230783 "" "Prometheus/2.43.0" 2024-09-10T19:07:32.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-mon[29577]: pgmap v369: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:07:32.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:32.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:32.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:32.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-mon[29577]: osdmap e272: 8 total, 8 up, 8 in 2024-09-10T19:07:32.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39"}]: dispatch 2024-09-10T19:07:32.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39"}]: dispatch 2024-09-10T19:07:32.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-mon[40674]: pgmap v369: 228 pgs: 32 creating+peering, 196 active+clean; 4.6 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:07:32.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:32.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:32.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:07:32.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-mon[40674]: osdmap e272: 8 total, 8 up, 8 in 2024-09-10T19:07:32.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39"}]: dispatch 2024-09-10T19:07:32.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39"}]: dispatch 2024-09-10T19:07:33.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:33 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:33.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39"}]': finished 2024-09-10T19:07:33.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:33 smithi194 ceph-mon[32464]: osdmap e273: 8 total, 8 up, 8 in 2024-09-10T19:07:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:33 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39"}]': finished 2024-09-10T19:07:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:33 smithi119 ceph-mon[29577]: osdmap e273: 8 total, 8 up, 8 in 2024-09-10T19:07:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:33 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:07:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-39"}]': finished 2024-09-10T19:07:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:33 smithi119 ceph-mon[40674]: osdmap e273: 8 total, 8 up, 8 in 2024-09-10T19:07:33.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:33.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:34.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-10T19:07:34.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-10T19:07:34.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-10T19:07:34.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: seed 79782 2024-09-10T19:07:34.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-10T19:07:34.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (223 ms) 2024-09-10T19:07:34.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-10T19:07:34.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10741 ms) 2024-09-10T19:07:34.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-10T19:07:34.230 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (35 ms) 2024-09-10T19:07:34.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (10999 ms total) 2024-09-10T19:07:34.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: 2024-09-10T19:07:34.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-10T19:07:34.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-10T19:07:34.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7035 ms) 2024-09-10T19:07:34.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-10T19:07:34.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7983 ms) 2024-09-10T19:07:34.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-10T19:07:34.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10224 ms) 2024-09-10T19:07:34.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-10T19:07:34.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: my_snaps [4] 2024-09-10T19:07:34.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: my_snaps [5,4] 2024-09-10T19:07:34.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: my_snaps [6,5,4] 2024-09-10T19:07:34.231 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: my_snaps [7,6,5,4] 2024-09-10T19:07:34.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: promoting some heads 2024-09-10T19:07:34.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: promoting from clones for snap 7 2024-09-10T19:07:34.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-10T19:07:34.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-10T19:07:34.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-10T19:07:34.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: waiting for scrubs... 2024-09-10T19:07:34.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: done waiting 2024-09-10T19:07:34.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (45762 ms) 2024-09-10T19:07:34.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-10T19:07:34.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10030 ms) 2024-09-10T19:07:34.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-10T19:07:34.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8229 ms) 2024-09-10T19:07:34.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-10T19:07:34.232 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (7817 ms) 2024-09-10T19:07:34.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-10T19:07:34.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (8030 ms) 2024-09-10T19:07:34.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-10T19:07:34.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10041 ms) 2024-09-10T19:07:34.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-10T19:07:34.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9114 ms) 2024-09-10T19:07:34.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-10T19:07:34.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (9832 ms) 2024-09-10T19:07:34.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-10T19:07:34.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12952 ms) 2024-09-10T19:07:34.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-10T19:07:34.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7983 ms) 2024-09-10T19:07:34.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-10T19:07:34.233 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8061 ms) 2024-09-10T19:07:34.234 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-10T19:07:34.234 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13035 ms) 2024-09-10T19:07:34.234 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-10T19:07:34.234 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7909 ms) 2024-09-10T19:07:34.234 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-10T19:07:34.234 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8016 ms) 2024-09-10T19:07:34.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:34 smithi194 ceph-mon[32464]: pgmap v372: 236 pgs: 236 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.9 MiB/s wr, 81 op/s 2024-09-10T19:07:34.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]': finished 2024-09-10T19:07:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:34 smithi194 ceph-mon[32464]: osdmap e274: 8 total, 8 up, 8 in 2024-09-10T19:07:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:34 smithi119 ceph-mon[29577]: pgmap v372: 236 pgs: 236 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.9 MiB/s wr, 81 op/s 2024-09-10T19:07:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]': finished 2024-09-10T19:07:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:34 smithi119 ceph-mon[29577]: osdmap e274: 8 total, 8 up, 8 in 2024-09-10T19:07:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:34 smithi119 ceph-mon[40674]: pgmap v372: 236 pgs: 236 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.9 MiB/s wr, 81 op/s 2024-09-10T19:07:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi119-79499-54"}]': finished 2024-09-10T19:07:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:34 smithi119 ceph-mon[40674]: osdmap e274: 8 total, 8 up, 8 in 2024-09-10T19:07:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3089046297' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]: dispatch 2024-09-10T19:07:36.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:36 smithi194 ceph-mon[32464]: pgmap v375: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-09-10T19:07:36.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]': finished 2024-09-10T19:07:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:36 smithi194 ceph-mon[32464]: osdmap e275: 8 total, 8 up, 8 in 2024-09-10T19:07:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:36.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi119-79499-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[40674]: pgmap v375: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-09-10T19:07:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]': finished 2024-09-10T19:07:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[40674]: osdmap e275: 8 total, 8 up, 8 in 2024-09-10T19:07:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi119-79499-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[29577]: pgmap v375: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-09-10T19:07:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi119-79499-54"}]': finished 2024-09-10T19:07:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[29577]: osdmap e275: 8 total, 8 up, 8 in 2024-09-10T19:07:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:36.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi119-79499-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:37.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi119-79499-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:37 smithi194 ceph-mon[32464]: osdmap e276: 8 total, 8 up, 8 in 2024-09-10T19:07:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi119-79499-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi119-79499-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:37 smithi119 ceph-mon[29577]: osdmap e276: 8 total, 8 up, 8 in 2024-09-10T19:07:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi119-79499-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:37.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:37.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi119-79499-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:37.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:37 smithi119 ceph-mon[40674]: osdmap e276: 8 total, 8 up, 8 in 2024-09-10T19:07:37.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi119-79499-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:37.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:37.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:38.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:07:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:07:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:38 smithi194 ceph-mon[32464]: pgmap v378: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:38 smithi194 ceph-mon[32464]: osdmap e277: 8 total, 8 up, 8 in 2024-09-10T19:07:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi119-79499-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi119-79499-55"}]': finished 2024-09-10T19:07:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:07:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:38 smithi194 ceph-mon[32464]: osdmap e278: 8 total, 8 up, 8 in 2024-09-10T19:07:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:38.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[29577]: pgmap v378: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[29577]: osdmap e277: 8 total, 8 up, 8 in 2024-09-10T19:07:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi119-79499-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi119-79499-55"}]': finished 2024-09-10T19:07:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:07:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[29577]: osdmap e278: 8 total, 8 up, 8 in 2024-09-10T19:07:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:38.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[40674]: pgmap v378: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[40674]: osdmap e277: 8 total, 8 up, 8 in 2024-09-10T19:07:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:07:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi119-79499-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi119-79499-55"}]': finished 2024-09-10T19:07:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:07:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[40674]: osdmap e278: 8 total, 8 up, 8 in 2024-09-10T19:07:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:39.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:39 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:07:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:39 smithi194 ceph-mon[32464]: osdmap e279: 8 total, 8 up, 8 in 2024-09-10T19:07:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:39 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:07:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:07:39.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:39 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:39.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:07:39.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:39 smithi119 ceph-mon[29577]: osdmap e279: 8 total, 8 up, 8 in 2024-09-10T19:07:39.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:39 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:07:39.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:07:39.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:39 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:39.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:07:39.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:39 smithi119 ceph-mon[40674]: osdmap e279: 8 total, 8 up, 8 in 2024-09-10T19:07:39.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:39 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:07:39.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:07:40.486 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:07:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:07:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:07:40.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:40 smithi194 ceph-mon[32464]: pgmap v381: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:40.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:40 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:40.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:40 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:07:40.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:40 smithi194 ceph-mon[32464]: osdmap e280: 8 total, 8 up, 8 in 2024-09-10T19:07:40.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:40 smithi119 ceph-mon[29577]: pgmap v381: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:40 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:40 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:07:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:40 smithi119 ceph-mon[29577]: osdmap e280: 8 total, 8 up, 8 in 2024-09-10T19:07:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:40.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:40 smithi119 ceph-mon[40674]: pgmap v381: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:40.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:40 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:40.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:40 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:07:40.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:40 smithi119 ceph-mon[40674]: osdmap e280: 8 total, 8 up, 8 in 2024-09-10T19:07:40.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:42.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:42 smithi194 ceph-mon[32464]: pgmap v384: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:42.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi119-79499-55"}]': finished 2024-09-10T19:07:42.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:42 smithi194 ceph-mon[32464]: osdmap e281: 8 total, 8 up, 8 in 2024-09-10T19:07:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41"}]: dispatch 2024-09-10T19:07:42.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41"}]: dispatch 2024-09-10T19:07:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[29577]: pgmap v384: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi119-79499-55"}]': finished 2024-09-10T19:07:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[29577]: osdmap e281: 8 total, 8 up, 8 in 2024-09-10T19:07:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41"}]: dispatch 2024-09-10T19:07:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41"}]: dispatch 2024-09-10T19:07:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[40674]: pgmap v384: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:42.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi119-79499-55"}]': finished 2024-09-10T19:07:42.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[40674]: osdmap e281: 8 total, 8 up, 8 in 2024-09-10T19:07:42.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi119-79499-55"}]: dispatch 2024-09-10T19:07:42.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:42.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:42.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41"}]: dispatch 2024-09-10T19:07:42.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41"}]: dispatch 2024-09-10T19:07:43.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:07:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:07:42] "GET /metrics HTTP/1.1" 200 230783 "" "Prometheus/2.43.0" 2024-09-10T19:07:43.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi119-79499-55"}]': finished 2024-09-10T19:07:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41"}]': finished 2024-09-10T19:07:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: osdmap e282: 8 total, 8 up, 8 in 2024-09-10T19:07:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi119-79499-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi119-79499-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:43.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:43.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:43.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:43.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:43.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:43 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:43.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi119-79499-55"}]': finished 2024-09-10T19:07:43.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41"}]': finished 2024-09-10T19:07:43.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: osdmap e282: 8 total, 8 up, 8 in 2024-09-10T19:07:43.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:43.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:43.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:43.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi119-79499-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi119-79499-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:43.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:43.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2263407132' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi119-79499-55"}]': finished 2024-09-10T19:07:43.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-41"}]': finished 2024-09-10T19:07:43.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: osdmap e282: 8 total, 8 up, 8 in 2024-09-10T19:07:43.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:43.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:43.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:43.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:43.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi119-79499-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:43.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi119-79499-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:43.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:43.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:43.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:43.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:43.569 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:43.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:43.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:43.570 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:43 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:44.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:44 smithi194 ceph-mon[32464]: pgmap v387: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:44.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi119-79499-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:44.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:44 smithi194 ceph-mon[32464]: osdmap e283: 8 total, 8 up, 8 in 2024-09-10T19:07:44.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi119-79499-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:44.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi119-79499-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:44.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:44 smithi119 ceph-mon[29577]: pgmap v387: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:44.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi119-79499-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:44.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:44 smithi119 ceph-mon[29577]: osdmap e283: 8 total, 8 up, 8 in 2024-09-10T19:07:44.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi119-79499-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:44.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi119-79499-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:44.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:44 smithi119 ceph-mon[40674]: pgmap v387: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:44.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi119-79499-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:44.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:44 smithi119 ceph-mon[40674]: osdmap e283: 8 total, 8 up, 8 in 2024-09-10T19:07:44.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi119-79499-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:44.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi119-79499-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:45.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:45.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:45 smithi119 ceph-mon[29577]: osdmap e284: 8 total, 8 up, 8 in 2024-09-10T19:07:45.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:45 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:45.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:45.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:45 smithi119 ceph-mon[40674]: osdmap e284: 8 total, 8 up, 8 in 2024-09-10T19:07:45.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:45 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:45.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:45.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:45 smithi194 ceph-mon[32464]: osdmap e284: 8 total, 8 up, 8 in 2024-09-10T19:07:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:45 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[29577]: pgmap v390: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi119-79499-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi119-79499-56"}]': finished 2024-09-10T19:07:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[29577]: osdmap e285: 8 total, 8 up, 8 in 2024-09-10T19:07:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi119-79782-6","var": "pg_num","format": "json"}]: dispatch 2024-09-10T19:07:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[40674]: pgmap v390: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi119-79499-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi119-79499-56"}]': finished 2024-09-10T19:07:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[40674]: osdmap e285: 8 total, 8 up, 8 in 2024-09-10T19:07:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi119-79782-6","var": "pg_num","format": "json"}]: dispatch 2024-09-10T19:07:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:46.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:46.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:46 smithi194 ceph-mon[32464]: pgmap v390: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:46 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi119-79499-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi119-79499-56"}]': finished 2024-09-10T19:07:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:46 smithi194 ceph-mon[32464]: osdmap e285: 8 total, 8 up, 8 in 2024-09-10T19:07:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi119-79782-6","var": "pg_num","format": "json"}]: dispatch 2024-09-10T19:07:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:47 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:47 smithi119 ceph-mon[29577]: osdmap e286: 8 total, 8 up, 8 in 2024-09-10T19:07:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_count","val": "8"}]: dispatch 2024-09-10T19:07:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:47 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_count","val": "8"}]: dispatch 2024-09-10T19:07:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:47.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:47 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:47.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:47 smithi119 ceph-mon[40674]: osdmap e286: 8 total, 8 up, 8 in 2024-09-10T19:07:47.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_count","val": "8"}]: dispatch 2024-09-10T19:07:47.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:47 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_count","val": "8"}]: dispatch 2024-09-10T19:07:47.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:47.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:47.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:47 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:47 smithi194 ceph-mon[32464]: osdmap e286: 8 total, 8 up, 8 in 2024-09-10T19:07:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_count","val": "8"}]: dispatch 2024-09-10T19:07:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:47 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_count","val": "8"}]: dispatch 2024-09-10T19:07:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:07:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:07:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:48 smithi119 ceph-mon[29577]: pgmap v393: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_count","val": "8"}]': finished 2024-09-10T19:07:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:48 smithi119 ceph-mon[29577]: osdmap e287: 8 total, 8 up, 8 in 2024-09-10T19:07:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:48.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:48 smithi119 ceph-mon[40674]: pgmap v393: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_count","val": "8"}]': finished 2024-09-10T19:07:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:48 smithi119 ceph-mon[40674]: osdmap e287: 8 total, 8 up, 8 in 2024-09-10T19:07:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:48.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:48.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:07:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:07:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:48 smithi194 ceph-mon[32464]: pgmap v393: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_count","val": "8"}]': finished 2024-09-10T19:07:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:48 smithi194 ceph-mon[32464]: osdmap e287: 8 total, 8 up, 8 in 2024-09-10T19:07:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:07:49.518 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-09-10T19:07:49.518 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-10T19:07:49.518 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: ok, hit_set contains 192:602f83fe:::foo:head 2024-09-10T19:07:49.518 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9031 ms) 2024-09-10T19:07:49.519 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-10T19:07:49.519 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg_num = 32 2024-09-10T19:07:49.519 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 0 ls 1725995270,0 2024-09-10T19:07:49.519 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 1 ls 1725995270,0 2024-09-10T19:07:49.519 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 2 ls 1725995270,0 2024-09-10T19:07:49.519 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 3 ls 1725995270,0 2024-09-10T19:07:49.519 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 4 ls 1725995270,0 2024-09-10T19:07:49.519 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 5 ls 1725995270,0 2024-09-10T19:07:49.519 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 6 ls 1725995270,0 2024-09-10T19:07:49.519 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 7 ls 1725995270,0 2024-09-10T19:07:49.519 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 8 ls 1725995270,0 2024-09-10T19:07:49.519 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 9 ls 1725995270,0 2024-09-10T19:07:49.519 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 10 ls 1725995270,0 2024-09-10T19:07:49.520 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 11 ls 1725995270,0 2024-09-10T19:07:49.520 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 12 ls 1725995270,0 2024-09-10T19:07:49.520 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 13 ls 1725995270,0 2024-09-10T19:07:49.520 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 14 ls 1725995270,0 2024-09-10T19:07:49.520 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 15 ls 1725995270,0 2024-09-10T19:07:49.520 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 16 ls 1725995270,0 2024-09-10T19:07:49.520 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 17 ls 1725995270,0 2024-09-10T19:07:49.520 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 18 ls 1725995270,0 2024-09-10T19:07:49.520 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 19 ls 1725995270,0 2024-09-10T19:07:49.520 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 20 ls 1725995270,0 2024-09-10T19:07:49.520 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 21 ls 1725995270,0 2024-09-10T19:07:49.520 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 22 ls 1725995270,0 2024-09-10T19:07:49.520 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 23 ls 1725995270,0 2024-09-10T19:07:49.520 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 24 ls 1725995270,0 2024-09-10T19:07:49.521 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 25 ls 1725995270,0 2024-09-10T19:07:49.521 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 26 ls 1725995270,0 2024-09-10T19:07:49.521 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 27 ls 1725995270,0 2024-09-10T19:07:49.521 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 28 ls 1725995270,0 2024-09-10T19:07:49.521 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 29 ls 1725995270,0 2024-09-10T19:07:49.521 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 30 ls 1725995270,0 2024-09-10T19:07:49.521 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg 31 ls 1725995270,0 2024-09-10T19:07:49.521 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: pg_num = 32 2024-09-10T19:07:49.521 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:6cac518f:::0:head 2024-09-10T19:07:49.521 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:02547ec2:::1:head 2024-09-10T19:07:49.521 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:f905c69b:::2:head 2024-09-10T19:07:49.521 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:cfc208b3:::3:head 2024-09-10T19:07:49.521 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:d83876eb:::4:head 2024-09-10T19:07:49.522 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:b29083e3:::5:head 2024-09-10T19:07:49.522 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:c4fdafeb:::6:head 2024-09-10T19:07:49.522 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:5c6b0b28:::7:head 2024-09-10T19:07:49.522 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:bd63b0f1:::8:head 2024-09-10T19:07:49.522 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:e960b815:::9:head 2024-09-10T19:07:49.522 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:52ea6a34:::10:head 2024-09-10T19:07:49.522 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:89d3ae78:::11:head 2024-09-10T19:07:49.522 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:de5d7c5f:::12:head 2024-09-10T19:07:49.522 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:566253c9:::13:head 2024-09-10T19:07:49.522 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:62a1935d:::14:head 2024-09-10T19:07:49.522 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:863748b0:::15:head 2024-09-10T19:07:49.522 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:3958e169:::16:head 2024-09-10T19:07:49.522 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:4d4dabf9:::17:head 2024-09-10T19:07:49.523 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:8391935d:::18:head 2024-09-10T19:07:49.523 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:28883081:::19:head 2024-09-10T19:07:49.523 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:69259c59:::20:head 2024-09-10T19:07:49.523 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:4bdb80b7:::21:head 2024-09-10T19:07:49.523 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:a11c5d71:::22:head 2024-09-10T19:07:49.523 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:271af37b:::23:head 2024-09-10T19:07:49.523 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:95b121be:::24:head 2024-09-10T19:07:49.523 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:58d1031b:::25:head 2024-09-10T19:07:49.523 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:0a050783:::26:head 2024-09-10T19:07:49.523 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:c709704c:::27:head 2024-09-10T19:07:49.523 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:cbe56eaf:::28:head 2024-09-10T19:07:49.523 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:86b4b162:::29:head 2024-09-10T19:07:49.523 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:70d89383:::30:head 2024-09-10T19:07:49.524 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:dd450c7c:::31:head 2024-09-10T19:07:49.524 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:6d5729b1:::32:head 2024-09-10T19:07:49.524 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:c388f3fb:::33:head 2024-09-10T19:07:49.524 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:56cfea31:::34:head 2024-09-10T19:07:49.524 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:9dbc1bf7:::35:head 2024-09-10T19:07:49.524 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:40b74ccd:::36:head 2024-09-10T19:07:49.524 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:4d5aaf42:::37:head 2024-09-10T19:07:49.524 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:920f362c:::38:head 2024-09-10T19:07:49.524 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:6cc53222:::39:head 2024-09-10T19:07:49.524 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:9cad833f:::40:head 2024-09-10T19:07:49.524 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:1ea84d41:::41:head 2024-09-10T19:07:49.524 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:c4480ef6:::42:head 2024-09-10T19:07:49.524 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:a694361e:::43:head 2024-09-10T19:07:49.524 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:d1bd33e9:::44:head 2024-09-10T19:07:49.525 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:ddc2cd5d:::45:head 2024-09-10T19:07:49.525 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:2b782207:::46:head 2024-09-10T19:07:49.525 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:7b187fca:::47:head 2024-09-10T19:07:49.525 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:90ecdf6f:::48:head 2024-09-10T19:07:49.525 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:a5ed95fe:::49:head 2024-09-10T19:07:49.525 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:ea0eaa55:::50:head 2024-09-10T19:07:49.525 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:f33ef17b:::51:head 2024-09-10T19:07:49.525 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:a0d1b2f6:::52:head 2024-09-10T19:07:49.525 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:60c5229e:::53:head 2024-09-10T19:07:49.525 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:edcbc575:::54:head 2024-09-10T19:07:49.525 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:102cf253:::55:head 2024-09-10T19:07:49.525 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:efb7fb0b:::56:head 2024-09-10T19:07:49.525 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:50d0a326:::57:head 2024-09-10T19:07:49.525 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:d4dc5daf:::58:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:3a130462:::59:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:ec87ed71:::60:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:d5bc9454:::61:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:3ddfe313:::62:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:7c2816b9:::63:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:47e00e4d:::64:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:c6410c18:::65:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:b48ed237:::66:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:cd63ad31:::67:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:b179e92b:::68:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:0d9f741a:::69:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:6d3352ae:::70:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:c6d5c19e:::71:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:bc4729c3:::72:head 2024-09-10T19:07:49.526 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:77e930b9:::73:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:0abeecfd:::74:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:b7c37e15:::75:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:b6378398:::76:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:02bd68de:::77:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:cc795d2d:::78:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:630d4fea:::79:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:e0d29ef5:::80:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:fd6f13d2:::81:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:606461d5:::82:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:eadbdc43:::83:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:8761d0bb:::84:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:9ef0186f:::85:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:e0d41294:::86:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:961de695:::87:head 2024-09-10T19:07:49.527 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:1423148f:::88:head 2024-09-10T19:07:49.528 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:633a8fa2:::89:head 2024-09-10T19:07:49.528 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:a8653809:::90:head 2024-09-10T19:07:49.528 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:3dac8b33:::91:head 2024-09-10T19:07:49.528 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:35aad435:::92:head 2024-09-10T19:07:49.528 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:f6dcc343:::93:head 2024-09-10T19:07:49.528 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:dbbdad87:::94:head 2024-09-10T19:07:49.528 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:1cb48ce0:::95:head 2024-09-10T19:07:49.528 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:03cd461c:::96:head 2024-09-10T19:07:49.528 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:17a4ea99:::97:head 2024-09-10T19:07:49.528 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:9993c9a7:::98:head 2024-09-10T19:07:49.528 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:6394211c:::99:head 2024-09-10T19:07:49.528 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:94c7ae57:::100:head 2024-09-10T19:07:49.528 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:6fdee5bb:::101:head 2024-09-10T19:07:49.529 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:9a477fd1:::102:head 2024-09-10T19:07:49.529 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:eb850916:::103:head 2024-09-10T19:07:49.529 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:affc56b9:::104:head 2024-09-10T19:07:49.529 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:b42dc814:::105:head 2024-09-10T19:07:49.529 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:f319f8f0:::106:head 2024-09-10T19:07:49.529 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:9a40b9de:::107:head 2024-09-10T19:07:49.529 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:8b524f28:::108:head 2024-09-10T19:07:49.529 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:e3de589f:::109:head 2024-09-10T19:07:49.529 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:90f90a5b:::110:head 2024-09-10T19:07:49.529 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:a7b4f1d7:::111:head 2024-09-10T19:07:49.529 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:af51766e:::112:head 2024-09-10T19:07:49.529 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:b6f90bd1:::113:head 2024-09-10T19:07:49.529 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:e0261208:::114:head 2024-09-10T19:07:49.529 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:c9569ef7:::115:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:61bebe50:::116:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:fe93412b:::117:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:d3d38bee:::118:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:3100ba0c:::119:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:d0560ada:::120:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:f0ea8b35:::121:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:766f231a:::122:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:a07a2582:::123:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:bd7c6b3a:::124:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:fb2ddaff:::125:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:4408e1fe:::126:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:ee1df7a7:::127:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:c3002909:::128:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:4f48ffa9:::129:head 2024-09-10T19:07:49.530 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:edf38733:::130:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:c08425c0:::131:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:5f902d98:::132:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:41ea2c93:::133:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:813cee13:::134:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:0131818d:::135:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:26ba5a85:::136:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:381b8a5a:::137:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:28797e47:::138:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:bfca7f22:::139:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:36807075:::140:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:80b03975:::141:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:5c15709b:::142:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:f39ea15e:::143:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:ea992956:::144:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:48887b1c:::145:head 2024-09-10T19:07:49.531 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:9f24a9dd:::146:head 2024-09-10T19:07:49.532 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:987f100b:::147:head 2024-09-10T19:07:49.532 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:d2dd3581:::148:head 2024-09-10T19:07:49.532 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:7fed1808:::149:head 2024-09-10T19:07:49.532 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:c80b70e9:::150:head 2024-09-10T19:07:49.532 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:85ed90f9:::151:head 2024-09-10T19:07:49.532 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:36428b24:::152:head 2024-09-10T19:07:49.532 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:d044c34a:::153:head 2024-09-10T19:07:49.532 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:7c18bf58:::154:head 2024-09-10T19:07:49.532 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:d1c21232:::155:head 2024-09-10T19:07:49.532 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:a7a3c575:::156:head 2024-09-10T19:07:49.532 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:87da0633:::157:head 2024-09-10T19:07:49.532 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:d5ac3822:::158:head 2024-09-10T19:07:49.532 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:3f20522d:::159:head 2024-09-10T19:07:49.532 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:6ca26563:::160:head 2024-09-10T19:07:49.533 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:532ce135:::161:head 2024-09-10T19:07:49.533 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:c78863e6:::162:head 2024-09-10T19:07:49.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]': finished 2024-09-10T19:07:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:07:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[29577]: osdmap e288: 8 total, 8 up, 8 in 2024-09-10T19:07:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-09-10T19:07:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-09-10T19:07:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:49.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:49.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]': finished 2024-09-10T19:07:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:07:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[40674]: osdmap e288: 8 total, 8 up, 8 in 2024-09-10T19:07:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-09-10T19:07:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-09-10T19:07:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:49.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:49 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:49.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi119-79499-56"}]': finished 2024-09-10T19:07:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:07:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:49 smithi194 ceph-mon[32464]: osdmap e288: 8 total, 8 up, 8 in 2024-09-10T19:07:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-09-10T19:07:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1637079883' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-09-10T19:07:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]: dispatch 2024-09-10T19:07:49.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:49 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:07:50.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:07:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:07:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: pgmap v396: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-09-10T19:07:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]': finished 2024-09-10T19:07:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: osdmap e289: 8 total, 8 up, 8 in 2024-09-10T19:07:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi119-79499-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi119-79499-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:50.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi119-79782-43","var": "pg_num","format": "json"}]: dispatch 2024-09-10T19:07:50.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:50.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:50.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43"}]: dispatch 2024-09-10T19:07:50.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:50 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43"}]: dispatch 2024-09-10T19:07:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: pgmap v396: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-09-10T19:07:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]': finished 2024-09-10T19:07:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: osdmap e289: 8 total, 8 up, 8 in 2024-09-10T19:07:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi119-79499-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi119-79499-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi119-79782-43","var": "pg_num","format": "json"}]: dispatch 2024-09-10T19:07:50.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:50.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:50.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43"}]: dispatch 2024-09-10T19:07:50.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43"}]: dispatch 2024-09-10T19:07:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: pgmap v396: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-09-10T19:07:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi119-79499-56"}]': finished 2024-09-10T19:07:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: osdmap e289: 8 total, 8 up, 8 in 2024-09-10T19:07:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:50.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:50.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:50.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:50.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi119-79499-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:50.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi119-79499-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:50.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi119-79782-43","var": "pg_num","format": "json"}]: dispatch 2024-09-10T19:07:50.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:50.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:07:50.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43"}]: dispatch 2024-09-10T19:07:50.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:50 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43"}]: dispatch 2024-09-10T19:07:51.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi119-79499-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:51.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43"}]': finished 2024-09-10T19:07:51.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:51 smithi194 ceph-mon[32464]: osdmap e290: 8 total, 8 up, 8 in 2024-09-10T19:07:51.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi119-79499-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:51.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi119-79499-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi119-79499-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43"}]': finished 2024-09-10T19:07:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:51 smithi119 ceph-mon[29577]: osdmap e290: 8 total, 8 up, 8 in 2024-09-10T19:07:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi119-79499-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi119-79499-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi119-79499-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-43"}]': finished 2024-09-10T19:07:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:51 smithi119 ceph-mon[40674]: osdmap e290: 8 total, 8 up, 8 in 2024-09-10T19:07:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi119-79499-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:51.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi119-79499-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:52 smithi119 ceph-mon[29577]: pgmap v399: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:52 smithi119 ceph-mon[29577]: osdmap e291: 8 total, 8 up, 8 in 2024-09-10T19:07:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:52 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:52.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:52 smithi119 ceph-mon[40674]: pgmap v399: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:52 smithi119 ceph-mon[40674]: osdmap e291: 8 total, 8 up, 8 in 2024-09-10T19:07:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:52 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:52.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:52 smithi194 ceph-mon[32464]: pgmap v399: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:52.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:52 smithi194 ceph-mon[32464]: osdmap e291: 8 total, 8 up, 8 in 2024-09-10T19:07:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:52 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:07:53.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:07:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:07:52] "GET /metrics HTTP/1.1" 200 230779 "" "Prometheus/2.43.0" 2024-09-10T19:07:53.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi119-79499-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi119-79499-57"}]': finished 2024-09-10T19:07:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:53 smithi194 ceph-mon[32464]: osdmap e292: 8 total, 8 up, 8 in 2024-09-10T19:07:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:53.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi119-79499-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi119-79499-57"}]': finished 2024-09-10T19:07:53.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:53 smithi119 ceph-mon[29577]: osdmap e292: 8 total, 8 up, 8 in 2024-09-10T19:07:53.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:53.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi119-79499-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi119-79499-57"}]': finished 2024-09-10T19:07:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:53 smithi119 ceph-mon[40674]: osdmap e292: 8 total, 8 up, 8 in 2024-09-10T19:07:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:53.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:07:54.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:54 smithi194 ceph-mon[32464]: pgmap v402: 236 pgs: 40 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:54 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:54 smithi194 ceph-mon[32464]: osdmap e293: 8 total, 8 up, 8 in 2024-09-10T19:07:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:54 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:54 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:54 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:54 smithi194 ceph-mon[32464]: osdmap e294: 8 total, 8 up, 8 in 2024-09-10T19:07:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:54 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:07:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:54 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:07:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:54 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:54 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[29577]: pgmap v402: 236 pgs: 40 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[29577]: osdmap e293: 8 total, 8 up, 8 in 2024-09-10T19:07:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[29577]: osdmap e294: 8 total, 8 up, 8 in 2024-09-10T19:07:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:07:54.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:07:54.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:54.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:54.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[40674]: pgmap v402: 236 pgs: 40 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:54.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:07:54.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[40674]: osdmap e293: 8 total, 8 up, 8 in 2024-09-10T19:07:54.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:54.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:07:54.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:07:54.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[40674]: osdmap e294: 8 total, 8 up, 8 in 2024-09-10T19:07:54.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:07:54.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:07:54.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:54.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:54 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:56.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:56 smithi194 ceph-mon[32464]: pgmap v405: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:56 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_count","val": "3"}]': finished 2024-09-10T19:07:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:56 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]': finished 2024-09-10T19:07:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:56 smithi194 ceph-mon[32464]: osdmap e295: 8 total, 8 up, 8 in 2024-09-10T19:07:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:07:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:56 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:07:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:56 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[29577]: pgmap v405: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_count","val": "3"}]': finished 2024-09-10T19:07:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]': finished 2024-09-10T19:07:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[29577]: osdmap e295: 8 total, 8 up, 8 in 2024-09-10T19:07:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:56.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:07:56.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:07:56.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:56.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[40674]: pgmap v405: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:07:56.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_count","val": "3"}]': finished 2024-09-10T19:07:56.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi119-79499-57"}]': finished 2024-09-10T19:07:56.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[40674]: osdmap e295: 8 total, 8 up, 8 in 2024-09-10T19:07:56.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1765253535' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:56.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:07:56.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:07:56.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:56 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]: dispatch 2024-09-10T19:07:57.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_period","val": "3"}]': finished 2024-09-10T19:07:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]': finished 2024-09-10T19:07:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:57 smithi194 ceph-mon[32464]: osdmap e296: 8 total, 8 up, 8 in 2024-09-10T19:07:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi119-79499-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi119-79499-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:57.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:07:57.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi119-79499-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:57.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:57 smithi194 ceph-mon[32464]: osdmap e297: 8 total, 8 up, 8 in 2024-09-10T19:07:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_period","val": "3"}]': finished 2024-09-10T19:07:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]': finished 2024-09-10T19:07:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[29577]: osdmap e296: 8 total, 8 up, 8 in 2024-09-10T19:07:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi119-79499-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi119-79499-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:07:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi119-79499-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:57.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[29577]: osdmap e297: 8 total, 8 up, 8 in 2024-09-10T19:07:57.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_period","val": "3"}]': finished 2024-09-10T19:07:57.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi119-79499-57"}]': finished 2024-09-10T19:07:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[40674]: osdmap e296: 8 total, 8 up, 8 in 2024-09-10T19:07:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:07:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi119-79499-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi119-79499-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:07:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:07:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi119-79499-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:07:57.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:57 smithi119 ceph-mon[40674]: osdmap e297: 8 total, 8 up, 8 in 2024-09-10T19:07:58.670 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:07:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:07:58.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:58 smithi194 ceph-mon[32464]: pgmap v408: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:58.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:07:58.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:07:58.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi119-79499-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:58.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi119-79499-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:58 smithi119 ceph-mon[40674]: pgmap v408: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:07:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:07:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi119-79499-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi119-79499-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:59.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:58 smithi119 ceph-mon[29577]: pgmap v408: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:07:59.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:07:59.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:07:59.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi119-79499-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:07:59.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi119-79499-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:08:00.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-10T19:08:00.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:59 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:59 smithi194 ceph-mon[32464]: osdmap e298: 8 total, 8 up, 8 in 2024-09-10T19:08:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi119-79499-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi119-79499-58"}]': finished 2024-09-10T19:08:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:07:59 smithi194 ceph-mon[32464]: osdmap e299: 8 total, 8 up, 8 in 2024-09-10T19:08:00.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-10T19:08:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:59 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:59 smithi119 ceph-mon[29577]: osdmap e298: 8 total, 8 up, 8 in 2024-09-10T19:08:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi119-79499-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi119-79499-58"}]': finished 2024-09-10T19:08:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:07:59 smithi119 ceph-mon[29577]: osdmap e299: 8 total, 8 up, 8 in 2024-09-10T19:08:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-10T19:08:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:59 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:59 smithi119 ceph-mon[40674]: osdmap e298: 8 total, 8 up, 8 in 2024-09-10T19:08:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi119-79499-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi119-79499-58"}]': finished 2024-09-10T19:08:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:07:59 smithi119 ceph-mon[40674]: osdmap e299: 8 total, 8 up, 8 in 2024-09-10T19:08:00.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:08:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:08:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:08:01.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:00 smithi194 ceph-mon[32464]: pgmap v411: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:01.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:00 smithi194 ceph-mon[32464]: osdmap e300: 8 total, 8 up, 8 in 2024-09-10T19:08:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:00 smithi119 ceph-mon[29577]: pgmap v411: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:00 smithi119 ceph-mon[29577]: osdmap e300: 8 total, 8 up, 8 in 2024-09-10T19:08:01.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:00 smithi119 ceph-mon[40674]: pgmap v411: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:01.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:00 smithi119 ceph-mon[40674]: osdmap e300: 8 total, 8 up, 8 in 2024-09-10T19:08:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:02 smithi119 ceph-mon[29577]: pgmap v414: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:02 smithi119 ceph-mon[29577]: osdmap e301: 8 total, 8 up, 8 in 2024-09-10T19:08:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:08:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:02 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:08:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:02 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:02.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:02 smithi119 ceph-mon[40674]: pgmap v414: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:02 smithi119 ceph-mon[40674]: osdmap e301: 8 total, 8 up, 8 in 2024-09-10T19:08:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:08:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:02 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:08:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:02 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:02.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:02 smithi194 ceph-mon[32464]: pgmap v414: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:02 smithi194 ceph-mon[32464]: osdmap e301: 8 total, 8 up, 8 in 2024-09-10T19:08:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:08:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:02 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:08:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:02 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:03.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:08:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:08:02] "GET /metrics HTTP/1.1" 200 228359 "" "Prometheus/2.43.0" 2024-09-10T19:08:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]': finished 2024-09-10T19:08:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:03 smithi194 ceph-mon[32464]: osdmap e302: 8 total, 8 up, 8 in 2024-09-10T19:08:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:08:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:08:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]': finished 2024-09-10T19:08:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:03 smithi119 ceph-mon[29577]: osdmap e302: 8 total, 8 up, 8 in 2024-09-10T19:08:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:08:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:08:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi119-79499-58"}]': finished 2024-09-10T19:08:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:03 smithi119 ceph-mon[40674]: osdmap e302: 8 total, 8 up, 8 in 2024-09-10T19:08:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3214339137' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:08:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]: dispatch 2024-09-10T19:08:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:04 smithi194 ceph-mon[32464]: pgmap v417: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:08:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]': finished 2024-09-10T19:08:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:04 smithi194 ceph-mon[32464]: osdmap e303: 8 total, 8 up, 8 in 2024-09-10T19:08:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:04.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:04.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:04.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[29577]: pgmap v417: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:08:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]': finished 2024-09-10T19:08:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[29577]: osdmap e303: 8 total, 8 up, 8 in 2024-09-10T19:08:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:04.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[40674]: pgmap v417: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:08:04.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi119-79499-58"}]': finished 2024-09-10T19:08:04.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[40674]: osdmap e303: 8 total, 8 up, 8 in 2024-09-10T19:08:04.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:04.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:04.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:04.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:04.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:04.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:05.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:05.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:05 smithi194 ceph-mon[32464]: osdmap e304: 8 total, 8 up, 8 in 2024-09-10T19:08:05.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi119-79499-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:05.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi119-79499-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:05.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:05.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:05 smithi119 ceph-mon[29577]: osdmap e304: 8 total, 8 up, 8 in 2024-09-10T19:08:05.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi119-79499-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:05.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi119-79499-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:05.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:05.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:05 smithi119 ceph-mon[40674]: osdmap e304: 8 total, 8 up, 8 in 2024-09-10T19:08:05.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi119-79499-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:05.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi119-79499-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:06.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:06 smithi194 ceph-mon[32464]: pgmap v420: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:08:06.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:06 smithi194 ceph-mon[32464]: osdmap e305: 8 total, 8 up, 8 in 2024-09-10T19:08:06.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:06 smithi119 ceph-mon[40674]: pgmap v420: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:08:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:06 smithi119 ceph-mon[40674]: osdmap e305: 8 total, 8 up, 8 in 2024-09-10T19:08:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:06 smithi119 ceph-mon[29577]: pgmap v420: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:08:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:06 smithi119 ceph-mon[29577]: osdmap e305: 8 total, 8 up, 8 in 2024-09-10T19:08:07.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi119-79499-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]': finished 2024-09-10T19:08:07.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:07 smithi194 ceph-mon[32464]: osdmap e306: 8 total, 8 up, 8 in 2024-09-10T19:08:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:07 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi119-79499-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]': finished 2024-09-10T19:08:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:07 smithi119 ceph-mon[29577]: osdmap e306: 8 total, 8 up, 8 in 2024-09-10T19:08:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:07 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi119-79499-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]': finished 2024-09-10T19:08:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:07 smithi119 ceph-mon[40674]: osdmap e306: 8 total, 8 up, 8 in 2024-09-10T19:08:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:07 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:08.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:08:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:08:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:08 smithi194 ceph-mon[32464]: pgmap v423: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:08 smithi194 ceph-mon[32464]: osdmap e307: 8 total, 8 up, 8 in 2024-09-10T19:08:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:08 smithi119 ceph-mon[29577]: pgmap v423: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:08 smithi119 ceph-mon[29577]: osdmap e307: 8 total, 8 up, 8 in 2024-09-10T19:08:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:08 smithi119 ceph-mon[40674]: pgmap v423: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:08 smithi119 ceph-mon[40674]: osdmap e307: 8 total, 8 up, 8 in 2024-09-10T19:08:09.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:09 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:09 smithi194 ceph-mon[32464]: osdmap e308: 8 total, 8 up, 8 in 2024-09-10T19:08:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:09 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:09 smithi119 ceph-mon[29577]: osdmap e308: 8 total, 8 up, 8 in 2024-09-10T19:08:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:09 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:09 smithi119 ceph-mon[40674]: osdmap e308: 8 total, 8 up, 8 in 2024-09-10T19:08:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:10.410 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:08:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:08:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:08:10.415 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:7462ddf6:::.RoundTripAppendPP (2729 ms) 2024-09-10T19:08:10.415 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-10T19:08:10.415 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3010 ms) 2024-09-10T19:08:10.416 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-10T19:08:10.416 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3031 ms) 2024-09-10T19:08:10.416 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-10T19:08:10.416 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3008 ms) 2024-09-10T19:08:10.416 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-10T19:08:10.416 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: setting pool EIO 2024-09-10T19:08:10.416 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: max_success 1426, min_failed 1427 2024-09-10T19:08:10.416 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4034 ms) 2024-09-10T19:08:10.416 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-10T19:08:10.416 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3014 ms) 2024-09-10T19:08:10.416 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (122028 ms total) 2024-09-10T19:08:10.416 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: 2024-09-10T19:08:10.416 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-10T19:08:10.416 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-10T19:08:10.417 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3008 ms) 2024-09-10T19:08:10.417 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-10T19:08:10.417 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9037 ms) 2024-09-10T19:08:10.417 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-10T19:08:10.417 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14889 ms) 2024-09-10T19:08:10.417 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-10T19:08:10.417 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3278 ms) 2024-09-10T19:08:10.417 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30213 ms total) 2024-09-10T19:08:10.417 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: 2024-09-10T19:08:10.417 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-10T19:08:10.417 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-10T19:08:10.417 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2780 ms) 2024-09-10T19:08:10.417 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2780 ms total) 2024-09-10T19:08:10.418 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: 2024-09-10T19:08:10.418 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-10T19:08:10.418 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-10T19:08:10.418 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13967 ms) 2024-09-10T19:08:10.418 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-10T19:08:10.418 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6976 ms) 2024-09-10T19:08:10.418 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-10T19:08:10.418 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7037 ms) 2024-09-10T19:08:10.418 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-10T19:08:10.418 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7032 ms) 2024-09-10T19:08:10.418 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-10T19:08:10.418 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3011 ms) 2024-09-10T19:08:10.418 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-10T19:08:10.418 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7013 ms) 2024-09-10T19:08:10.419 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-10T19:08:10.419 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6919 ms) 2024-09-10T19:08:10.419 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-10T19:08:10.419 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7004 ms) 2024-09-10T19:08:10.419 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-10T19:08:10.419 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7014 ms) 2024-09-10T19:08:10.419 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-10T19:08:10.419 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7069 ms) 2024-09-10T19:08:10.419 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-10T19:08:10.419 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7023 ms) 2024-09-10T19:08:10.419 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-10T19:08:10.419 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7021 ms) 2024-09-10T19:08:10.419 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-10T19:08:10.419 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7056 ms) 2024-09-10T19:08:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:10 smithi194 ceph-mon[32464]: pgmap v426: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-10T19:08:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]': finished 2024-09-10T19:08:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:10 smithi194 ceph-mon[32464]: osdmap e309: 8 total, 8 up, 8 in 2024-09-10T19:08:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45"}]: dispatch 2024-09-10T19:08:10.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45"}]: dispatch 2024-09-10T19:08:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[29577]: pgmap v426: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-10T19:08:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]': finished 2024-09-10T19:08:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[29577]: osdmap e309: 8 total, 8 up, 8 in 2024-09-10T19:08:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45"}]: dispatch 2024-09-10T19:08:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45"}]: dispatch 2024-09-10T19:08:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[40674]: pgmap v426: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-10T19:08:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi119-79499-59"}]': finished 2024-09-10T19:08:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[40674]: osdmap e309: 8 total, 8 up, 8 in 2024-09-10T19:08:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/4241904971' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]: dispatch 2024-09-10T19:08:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45"}]: dispatch 2024-09-10T19:08:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45"}]: dispatch 2024-09-10T19:08:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]': finished 2024-09-10T19:08:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45"}]': finished 2024-09-10T19:08:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:11 smithi194 ceph-mon[32464]: osdmap e310: 8 total, 8 up, 8 in 2024-09-10T19:08:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]': finished 2024-09-10T19:08:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45"}]': finished 2024-09-10T19:08:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:11 smithi119 ceph-mon[29577]: osdmap e310: 8 total, 8 up, 8 in 2024-09-10T19:08:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi119-79499-59"}]': finished 2024-09-10T19:08:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-45"}]': finished 2024-09-10T19:08:11.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:11 smithi119 ceph-mon[40674]: osdmap e310: 8 total, 8 up, 8 in 2024-09-10T19:08:12.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:12 smithi119 ceph-mon[29577]: pgmap v429: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-10T19:08:12.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:12 smithi119 ceph-mon[29577]: osdmap e311: 8 total, 8 up, 8 in 2024-09-10T19:08:12.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3003314689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:12.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:12.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:12 smithi119 ceph-mon[40674]: pgmap v429: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-10T19:08:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:12 smithi119 ceph-mon[40674]: osdmap e311: 8 total, 8 up, 8 in 2024-09-10T19:08:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3003314689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:12.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:12 smithi194 ceph-mon[32464]: pgmap v429: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-09-10T19:08:12.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:12 smithi194 ceph-mon[32464]: osdmap e311: 8 total, 8 up, 8 in 2024-09-10T19:08:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3003314689' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:13.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:08:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:08:12] "GET /metrics HTTP/1.1" 200 228359 "" "Prometheus/2.43.0" 2024-09-10T19:08:13.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:13 smithi194 ceph-mon[32464]: osdmap e312: 8 total, 8 up, 8 in 2024-09-10T19:08:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:13 smithi194 ceph-mon[32464]: osdmap e313: 8 total, 8 up, 8 in 2024-09-10T19:08:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:13 smithi119 ceph-mon[29577]: osdmap e312: 8 total, 8 up, 8 in 2024-09-10T19:08:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:13 smithi119 ceph-mon[29577]: osdmap e313: 8 total, 8 up, 8 in 2024-09-10T19:08:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi119-79499-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:13 smithi119 ceph-mon[40674]: osdmap e312: 8 total, 8 up, 8 in 2024-09-10T19:08:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:13.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:13.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:13.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:13 smithi119 ceph-mon[40674]: osdmap e313: 8 total, 8 up, 8 in 2024-09-10T19:08:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:14 smithi194 ceph-mon[32464]: pgmap v432: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:08:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:14 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi119-79499-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:14.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:14.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi119-79499-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:14.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:14.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:14 smithi194 ceph-mon[32464]: osdmap e314: 8 total, 8 up, 8 in 2024-09-10T19:08:14.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:14.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi119-79499-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:14.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[29577]: pgmap v432: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:08:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi119-79499-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi119-79499-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[29577]: osdmap e314: 8 total, 8 up, 8 in 2024-09-10T19:08:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi119-79499-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:14.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:14.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[40674]: pgmap v432: 260 pgs: 32 creating+peering, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:08:14.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:14.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:14.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:14.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi119-79499-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:14.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:14.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:14.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi119-79499-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:14.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:14.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[40674]: osdmap e314: 8 total, 8 up, 8 in 2024-09-10T19:08:14.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:14.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi119-79499-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:14.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:16.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:16 smithi194 ceph-mon[32464]: pgmap v435: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-47"}]': finished 2024-09-10T19:08:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:16 smithi194 ceph-mon[32464]: osdmap e315: 8 total, 8 up, 8 in 2024-09-10T19:08:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-47", "mode": "writeback"}]: dispatch 2024-09-10T19:08:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-47", "mode": "writeback"}]: dispatch 2024-09-10T19:08:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:16 smithi119 ceph-mon[29577]: pgmap v435: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-47"}]': finished 2024-09-10T19:08:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:16 smithi119 ceph-mon[29577]: osdmap e315: 8 total, 8 up, 8 in 2024-09-10T19:08:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-47", "mode": "writeback"}]: dispatch 2024-09-10T19:08:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-47", "mode": "writeback"}]: dispatch 2024-09-10T19:08:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:16 smithi119 ceph-mon[40674]: pgmap v435: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-47"}]': finished 2024-09-10T19:08:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:16 smithi119 ceph-mon[40674]: osdmap e315: 8 total, 8 up, 8 in 2024-09-10T19:08:16.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-47", "mode": "writeback"}]: dispatch 2024-09-10T19:08:16.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-47", "mode": "writeback"}]: dispatch 2024-09-10T19:08:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:17 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:08:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi119-79499-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi119-79499-61"}]': finished 2024-09-10T19:08:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-47", "mode": "writeback"}]': finished 2024-09-10T19:08:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:17 smithi194 ceph-mon[32464]: osdmap e316: 8 total, 8 up, 8 in 2024-09-10T19:08:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:08:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:08:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi119-79499-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi119-79499-61"}]': finished 2024-09-10T19:08:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-47", "mode": "writeback"}]': finished 2024-09-10T19:08:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[29577]: osdmap e316: 8 total, 8 up, 8 in 2024-09-10T19:08:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:08:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:08:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi119-79499-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi119-79499-61"}]': finished 2024-09-10T19:08:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-47", "mode": "writeback"}]': finished 2024-09-10T19:08:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[40674]: osdmap e316: 8 total, 8 up, 8 in 2024-09-10T19:08:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:08:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:18.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:18 smithi119 ceph-mon[29577]: pgmap v438: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:18.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:08:18.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:18 smithi119 ceph-mon[29577]: osdmap e317: 8 total, 8 up, 8 in 2024-09-10T19:08:18.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:18.719 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:18.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:18 smithi119 ceph-mon[40674]: pgmap v438: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:18.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:08:18.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:18 smithi119 ceph-mon[40674]: osdmap e317: 8 total, 8 up, 8 in 2024-09-10T19:08:18.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:18.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:18.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:08:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:08:18.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:18 smithi194 ceph-mon[32464]: pgmap v438: 236 pgs: 8 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:18.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:08:18.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:18 smithi194 ceph-mon[32464]: osdmap e317: 8 total, 8 up, 8 in 2024-09-10T19:08:18.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:18.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:19.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:19 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:08:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:19 smithi194 ceph-mon[32464]: osdmap e318: 8 total, 8 up, 8 in 2024-09-10T19:08:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:19 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi119-79499-61"}]': finished 2024-09-10T19:08:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:08:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:19 smithi194 ceph-mon[32464]: osdmap e319: 8 total, 8 up, 8 in 2024-09-10T19:08:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:08:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[29577]: osdmap e318: 8 total, 8 up, 8 in 2024-09-10T19:08:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:19.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi119-79499-61"}]': finished 2024-09-10T19:08:19.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:08:19.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[29577]: osdmap e319: 8 total, 8 up, 8 in 2024-09-10T19:08:19.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:08:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[40674]: osdmap e318: 8 total, 8 up, 8 in 2024-09-10T19:08:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:19.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi119-79499-61"}]': finished 2024-09-10T19:08:19.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:08:19.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[40674]: osdmap e319: 8 total, 8 up, 8 in 2024-09-10T19:08:19.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi119-79499-61"}]: dispatch 2024-09-10T19:08:20.450 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:08:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:08:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:08:20.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:20 smithi194 ceph-mon[32464]: pgmap v441: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T19:08:20.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:20 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:20.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:20.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:20.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi119-79499-61"}]': finished 2024-09-10T19:08:20.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:08:20.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:20 smithi194 ceph-mon[32464]: osdmap e320: 8 total, 8 up, 8 in 2024-09-10T19:08:20.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:08:20.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:08:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[29577]: pgmap v441: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T19:08:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi119-79499-61"}]': finished 2024-09-10T19:08:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:08:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[29577]: osdmap e320: 8 total, 8 up, 8 in 2024-09-10T19:08:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:08:20.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:08:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[40674]: pgmap v441: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T19:08:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/124398184' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi119-79499-61"}]': finished 2024-09-10T19:08:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:08:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[40674]: osdmap e320: 8 total, 8 up, 8 in 2024-09-10T19:08:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:08:20.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:08:21.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:21 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-10T19:08:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:21 smithi194 ceph-mon[32464]: osdmap e321: 8 total, 8 up, 8 in 2024-09-10T19:08:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi119-79499-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:08:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:21 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:08:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-10T19:08:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[29577]: osdmap e321: 8 total, 8 up, 8 in 2024-09-10T19:08:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi119-79499-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:08:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:08:21.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:21.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:21.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:21.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-10T19:08:21.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:21.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[40674]: osdmap e321: 8 total, 8 up, 8 in 2024-09-10T19:08:21.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi119-79499-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:21.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:08:21.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:21 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:08:23.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:22 smithi194 ceph-mon[32464]: pgmap v444: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T19:08:23.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:22 smithi119 ceph-mon[29577]: pgmap v444: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T19:08:23.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:22 smithi119 ceph-mon[40674]: pgmap v444: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T19:08:23.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:08:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:08:22] "GET /metrics HTTP/1.1" 200 230780 "" "Prometheus/2.43.0" 2024-09-10T19:08:24.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-10T19:08:24.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:23 smithi194 ceph-mon[32464]: osdmap e322: 8 total, 8 up, 8 in 2024-09-10T19:08:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:23 smithi194 ceph-mon[32464]: pgmap v447: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:08:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi119-79499-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi119-79499-62"}]': finished 2024-09-10T19:08:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:08:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:23 smithi194 ceph-mon[32464]: osdmap e323: 8 total, 8 up, 8 in 2024-09-10T19:08:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-10T19:08:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[29577]: osdmap e322: 8 total, 8 up, 8 in 2024-09-10T19:08:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[29577]: pgmap v447: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:08:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi119-79499-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi119-79499-62"}]': finished 2024-09-10T19:08:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:08:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[29577]: osdmap e323: 8 total, 8 up, 8 in 2024-09-10T19:08:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-10T19:08:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[40674]: osdmap e322: 8 total, 8 up, 8 in 2024-09-10T19:08:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[40674]: pgmap v447: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:08:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi119-79499-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi119-79499-62"}]': finished 2024-09-10T19:08:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:08:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[40674]: osdmap e323: 8 total, 8 up, 8 in 2024-09-10T19:08:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:24.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:25.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]': finished 2024-09-10T19:08:25.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:25 smithi194 ceph-mon[32464]: osdmap e324: 8 total, 8 up, 8 in 2024-09-10T19:08:25.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:25 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:25.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:25.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:25 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:25.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]': finished 2024-09-10T19:08:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:25 smithi119 ceph-mon[29577]: osdmap e324: 8 total, 8 up, 8 in 2024-09-10T19:08:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:25 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:25 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:25.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]': finished 2024-09-10T19:08:25.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:25 smithi119 ceph-mon[40674]: osdmap e324: 8 total, 8 up, 8 in 2024-09-10T19:08:25.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:25 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:25.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:25.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:25 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:25.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-47"}]: dispatch 2024-09-10T19:08:26.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:26 smithi194 ceph-mon[32464]: pgmap v450: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:08:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:26 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:26 smithi194 ceph-mon[32464]: osdmap e325: 8 total, 8 up, 8 in 2024-09-10T19:08:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:26.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:26 smithi119 ceph-mon[29577]: pgmap v450: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:08:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:26 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:26 smithi119 ceph-mon[29577]: osdmap e325: 8 total, 8 up, 8 in 2024-09-10T19:08:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:26 smithi119 ceph-mon[40674]: pgmap v450: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:08:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:26 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:26 smithi119 ceph-mon[40674]: osdmap e325: 8 total, 8 up, 8 in 2024-09-10T19:08:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:27.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62"}]': finished 2024-09-10T19:08:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:27 smithi194 ceph-mon[32464]: osdmap e326: 8 total, 8 up, 8 in 2024-09-10T19:08:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:27 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62"}]': finished 2024-09-10T19:08:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:27 smithi119 ceph-mon[29577]: osdmap e326: 8 total, 8 up, 8 in 2024-09-10T19:08:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:27 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:27.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi119-79499-62"}]': finished 2024-09-10T19:08:27.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:27 smithi119 ceph-mon[40674]: osdmap e326: 8 total, 8 up, 8 in 2024-09-10T19:08:27.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi119-79499-62"}]: dispatch 2024-09-10T19:08:27.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:27.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:27 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:28.670 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:08:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:08:28.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:28 smithi194 ceph-mon[32464]: pgmap v453: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:28.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi119-79499-62"}]': finished 2024-09-10T19:08:28.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:28.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:28 smithi194 ceph-mon[32464]: osdmap e327: 8 total, 8 up, 8 in 2024-09-10T19:08:28.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:28.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:28.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:28.671 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:28.671 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:28.671 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:28.671 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi119-79499-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:28.671 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi119-79499-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[29577]: pgmap v453: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi119-79499-62"}]': finished 2024-09-10T19:08:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[29577]: osdmap e327: 8 total, 8 up, 8 in 2024-09-10T19:08:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:28.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:28.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:28.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi119-79499-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:28.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi119-79499-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:28.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[40674]: pgmap v453: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:28.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2797876240' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi119-79499-62"}]': finished 2024-09-10T19:08:28.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:28.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[40674]: osdmap e327: 8 total, 8 up, 8 in 2024-09-10T19:08:28.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:28.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:28.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:28.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:28.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:28.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:28.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi119-79499-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:28.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi119-79499-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:29.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:29 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:29.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:29.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi119-79499-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:29.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:29 smithi194 ceph-mon[32464]: osdmap e328: 8 total, 8 up, 8 in 2024-09-10T19:08:29.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:29.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi119-79499-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:29.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:29.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi119-79499-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi119-79499-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[29577]: osdmap e328: 8 total, 8 up, 8 in 2024-09-10T19:08:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi119-79499-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi119-79499-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:29.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:29.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:29.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi119-79499-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:29.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[40674]: osdmap e328: 8 total, 8 up, 8 in 2024-09-10T19:08:29.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:29.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi119-79499-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:29.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:29.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi119-79499-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:30.476 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:08:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:08:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:08:30.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:30 smithi194 ceph-mon[32464]: pgmap v456: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:08:30.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:30 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-49"}]': finished 2024-09-10T19:08:30.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:30 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-49", "mode": "readproxy"}]: dispatch 2024-09-10T19:08:30.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:30 smithi194 ceph-mon[32464]: osdmap e329: 8 total, 8 up, 8 in 2024-09-10T19:08:30.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:30 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-49", "mode": "readproxy"}]: dispatch 2024-09-10T19:08:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:30 smithi119 ceph-mon[29577]: pgmap v456: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:08:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:30 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-49"}]': finished 2024-09-10T19:08:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:30 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-49", "mode": "readproxy"}]: dispatch 2024-09-10T19:08:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:30 smithi119 ceph-mon[29577]: osdmap e329: 8 total, 8 up, 8 in 2024-09-10T19:08:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:30 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-49", "mode": "readproxy"}]: dispatch 2024-09-10T19:08:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:30 smithi119 ceph-mon[40674]: pgmap v456: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:08:30.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:30 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-49"}]': finished 2024-09-10T19:08:30.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:30 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-49", "mode": "readproxy"}]: dispatch 2024-09-10T19:08:30.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:30 smithi119 ceph-mon[40674]: osdmap e329: 8 total, 8 up, 8 in 2024-09-10T19:08:30.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:30 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-49", "mode": "readproxy"}]: dispatch 2024-09-10T19:08:31.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:31 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:08:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi119-79499-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi119-79499-63"}]': finished 2024-09-10T19:08:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-49", "mode": "readproxy"}]': finished 2024-09-10T19:08:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:31 smithi194 ceph-mon[32464]: osdmap e330: 8 total, 8 up, 8 in 2024-09-10T19:08:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:31 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:08:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi119-79499-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi119-79499-63"}]': finished 2024-09-10T19:08:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-49", "mode": "readproxy"}]': finished 2024-09-10T19:08:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:31 smithi119 ceph-mon[29577]: osdmap e330: 8 total, 8 up, 8 in 2024-09-10T19:08:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:31 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:08:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi119-79499-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi119-79499-63"}]': finished 2024-09-10T19:08:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-49", "mode": "readproxy"}]': finished 2024-09-10T19:08:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:31 smithi119 ceph-mon[40674]: osdmap e330: 8 total, 8 up, 8 in 2024-09-10T19:08:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:32 smithi194 ceph-mon[32464]: pgmap v459: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:08:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:32 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:32 smithi194 ceph-mon[32464]: osdmap e331: 8 total, 8 up, 8 in 2024-09-10T19:08:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:32.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:08:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:08:32] "GET /metrics HTTP/1.1" 200 230797 "" "Prometheus/2.43.0" 2024-09-10T19:08:32.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:32 smithi119 ceph-mon[29577]: pgmap v459: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:08:32.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:32 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:32.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:32 smithi119 ceph-mon[29577]: osdmap e331: 8 total, 8 up, 8 in 2024-09-10T19:08:32.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:32.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:32 smithi119 ceph-mon[40674]: pgmap v459: 236 pgs: 8 unknown, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:08:32.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:32 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:32.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:32 smithi119 ceph-mon[40674]: osdmap e331: 8 total, 8 up, 8 in 2024-09-10T19:08:32.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:33.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:33 smithi194 ceph-mon[32464]: osdmap e332: 8 total, 8 up, 8 in 2024-09-10T19:08:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:33 smithi119 ceph-mon[29577]: osdmap e332: 8 total, 8 up, 8 in 2024-09-10T19:08:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:33 smithi119 ceph-mon[40674]: osdmap e332: 8 total, 8 up, 8 in 2024-09-10T19:08:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:34.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:34 smithi194 ceph-mon[32464]: pgmap v462: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:08:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]': finished 2024-09-10T19:08:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:34 smithi194 ceph-mon[32464]: osdmap e333: 8 total, 8 up, 8 in 2024-09-10T19:08:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:34 smithi119 ceph-mon[29577]: pgmap v462: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:08:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]': finished 2024-09-10T19:08:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:34 smithi119 ceph-mon[29577]: osdmap e333: 8 total, 8 up, 8 in 2024-09-10T19:08:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:34 smithi119 ceph-mon[40674]: pgmap v462: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:08:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi119-79499-63"}]': finished 2024-09-10T19:08:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:34 smithi119 ceph-mon[40674]: osdmap e333: 8 total, 8 up, 8 in 2024-09-10T19:08:34.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2828721410' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:34.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]: dispatch 2024-09-10T19:08:35.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:35 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]': finished 2024-09-10T19:08:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:35 smithi194 ceph-mon[32464]: osdmap e334: 8 total, 8 up, 8 in 2024-09-10T19:08:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi119-79499-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:35 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]': finished 2024-09-10T19:08:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:35 smithi119 ceph-mon[29577]: osdmap e334: 8 total, 8 up, 8 in 2024-09-10T19:08:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi119-79499-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:35 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi119-79499-63"}]': finished 2024-09-10T19:08:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:35 smithi119 ceph-mon[40674]: osdmap e334: 8 total, 8 up, 8 in 2024-09-10T19:08:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:35.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:35.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi119-79499-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:37.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:36 smithi194 ceph-mon[32464]: pgmap v465: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:08:37.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi119-79499-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:37.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:36 smithi194 ceph-mon[32464]: osdmap e335: 8 total, 8 up, 8 in 2024-09-10T19:08:37.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi119-79499-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:37.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:36 smithi119 ceph-mon[29577]: pgmap v465: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:08:37.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi119-79499-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:37.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:36 smithi119 ceph-mon[29577]: osdmap e335: 8 total, 8 up, 8 in 2024-09-10T19:08:37.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi119-79499-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:37.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:36 smithi119 ceph-mon[40674]: pgmap v465: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:08:37.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi119-79499-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:37.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:36 smithi119 ceph-mon[40674]: osdmap e335: 8 total, 8 up, 8 in 2024-09-10T19:08:37.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi119-79499-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:38.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:37 smithi119 ceph-mon[29577]: osdmap e336: 8 total, 8 up, 8 in 2024-09-10T19:08:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:37 smithi119 ceph-mon[29577]: pgmap v468: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:37 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi119-79499-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi119-79499-64"}]': finished 2024-09-10T19:08:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:37 smithi119 ceph-mon[29577]: osdmap e337: 8 total, 8 up, 8 in 2024-09-10T19:08:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:37 smithi119 ceph-mon[40674]: osdmap e336: 8 total, 8 up, 8 in 2024-09-10T19:08:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:37 smithi119 ceph-mon[40674]: pgmap v468: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:37 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi119-79499-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi119-79499-64"}]': finished 2024-09-10T19:08:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:37 smithi119 ceph-mon[40674]: osdmap e337: 8 total, 8 up, 8 in 2024-09-10T19:08:38.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:37 smithi194 ceph-mon[32464]: osdmap e336: 8 total, 8 up, 8 in 2024-09-10T19:08:38.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:37 smithi194 ceph-mon[32464]: pgmap v468: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:38.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:37 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:38.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi119-79499-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi119-79499-64"}]': finished 2024-09-10T19:08:38.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:37 smithi194 ceph-mon[32464]: osdmap e337: 8 total, 8 up, 8 in 2024-09-10T19:08:38.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:08:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:08:39.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:39 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:39.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:39 smithi119 ceph-mon[29577]: osdmap e338: 8 total, 8 up, 8 in 2024-09-10T19:08:39.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:39 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:39.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:39 smithi119 ceph-mon[40674]: osdmap e338: 8 total, 8 up, 8 in 2024-09-10T19:08:40.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:39 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:40.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:39 smithi194 ceph-mon[32464]: osdmap e338: 8 total, 8 up, 8 in 2024-09-10T19:08:40.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:08:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:08:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:08:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:40 smithi119 ceph-mon[29577]: pgmap v471: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:40 smithi119 ceph-mon[29577]: osdmap e339: 8 total, 8 up, 8 in 2024-09-10T19:08:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:40.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:40 smithi119 ceph-mon[40674]: pgmap v471: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:40.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:40 smithi119 ceph-mon[40674]: osdmap e339: 8 total, 8 up, 8 in 2024-09-10T19:08:40.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:40 smithi194 ceph-mon[32464]: pgmap v471: 236 pgs: 8 creating+peering, 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:40 smithi194 ceph-mon[32464]: osdmap e339: 8 total, 8 up, 8 in 2024-09-10T19:08:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:41.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi119-79499-64"}]': finished 2024-09-10T19:08:41.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[29577]: osdmap e340: 8 total, 8 up, 8 in 2024-09-10T19:08:41.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:41.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:41.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:41.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi119-79499-64"}]': finished 2024-09-10T19:08:41.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:08:41.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[29577]: osdmap e341: 8 total, 8 up, 8 in 2024-09-10T19:08:41.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi119-79499-64"}]': finished 2024-09-10T19:08:41.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[40674]: osdmap e340: 8 total, 8 up, 8 in 2024-09-10T19:08:41.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:41.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:41.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:41.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi119-79499-64"}]': finished 2024-09-10T19:08:41.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:08:41.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:41 smithi119 ceph-mon[40674]: osdmap e341: 8 total, 8 up, 8 in 2024-09-10T19:08:42.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi119-79499-64"}]': finished 2024-09-10T19:08:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:41 smithi194 ceph-mon[32464]: osdmap e340: 8 total, 8 up, 8 in 2024-09-10T19:08:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi119-79499-64"}]: dispatch 2024-09-10T19:08:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:41 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/666481861' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi119-79499-64"}]': finished 2024-09-10T19:08:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:41 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:08:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:41 smithi194 ceph-mon[32464]: osdmap e341: 8 total, 8 up, 8 in 2024-09-10T19:08:42.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:08:42] "GET /metrics HTTP/1.1" 200 230797 "" "Prometheus/2.43.0" 2024-09-10T19:08:42.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[29577]: pgmap v474: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:42.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:42.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:42.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:42.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:42.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:42.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:42.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi119-79499-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:42.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi119-79499-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:42.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:42.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[40674]: pgmap v474: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:42.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:42.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:42.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:42.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:42.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:42.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:42.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi119-79499-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:42.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi119-79499-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:42.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:42 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:43.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:42 smithi194 ceph-mon[32464]: pgmap v474: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi119-79499-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi119-79499-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:42 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:43.777 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullP163:head 2024-09-10T19:08:43.778 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:5d165639:::164:head 2024-09-10T19:08:43.778 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:f43765fc:::165:head 2024-09-10T19:08:43.778 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:b4c720e9:::166:head 2024-09-10T19:08:43.778 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:e694b040:::167:head 2024-09-10T19:08:43.778 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:afa38db2:::168:head 2024-09-10T19:08:43.778 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:77ba9f53:::169:head 2024-09-10T19:08:43.778 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:87495034:::170:head 2024-09-10T19:08:43.778 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:7c96bf0e:::171:head 2024-09-10T19:08:43.778 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:dbe346cc:::172:head 2024-09-10T19:08:43.778 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:e943ec24:::173:head 2024-09-10T19:08:43.779 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:f97a9c0c:::174:head 2024-09-10T19:08:43.779 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:6f26e74d:::175:head 2024-09-10T19:08:43.779 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:4f95e106:::176:head 2024-09-10T19:08:43.779 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:0e6f2f8f:::177:head 2024-09-10T19:08:43.779 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:05db05f1:::178:head 2024-09-10T19:08:43.779 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:38a78d66:::179:head 2024-09-10T19:08:43.779 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:d095610b:::180:head 2024-09-10T19:08:43.779 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:a1a9d709:::181:head 2024-09-10T19:08:43.779 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:1e5d39db:::182:head 2024-09-10T19:08:43.779 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:f7df4fb9:::183:head 2024-09-10T19:08:43.779 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:03a7f161:::184:head 2024-09-10T19:08:43.779 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:ba70721e:::185:head 2024-09-10T19:08:43.779 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:28e5662d:::186:head 2024-09-10T19:08:43.779 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:973d52de:::187:head 2024-09-10T19:08:43.780 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:4303eb1c:::188:head 2024-09-10T19:08:43.780 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:b990b48e:::189:head 2024-09-10T19:08:43.780 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:29b8165b:::190:head 2024-09-10T19:08:43.780 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:3547f197:::191:head 2024-09-10T19:08:43.780 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:7e260936:::192:head 2024-09-10T19:08:43.780 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:1abec7b1:::193:head 2024-09-10T19:08:43.780 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:10fdda93:::194:head 2024-09-10T19:08:43.780 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:15817eea:::195:head 2024-09-10T19:08:43.780 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:770bab57:::196:head 2024-09-10T19:08:43.780 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:ed9e13e7:::197:head 2024-09-10T19:08:43.780 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:71471a8f:::198:head 2024-09-10T19:08:43.780 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: checking for 194:10fb1d02:::199:head 2024-09-10T19:08:43.780 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8299 ms) 2024-09-10T19:08:43.781 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-10T19:08:43.781 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995279,0 2024-09-10T19:08:43.781 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: first is 1725995279 2024-09-10T19:08:43.781 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995279,0 2024-09-10T19:08:43.781 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995279,0 2024-09-10T19:08:43.781 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995279,1725995281,1725995282,0 2024-09-10T19:08:43.781 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995279,1725995281,1725995282,0 2024-09-10T19:08:43.781 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995279,1725995281,1725995282,0 2024-09-10T19:08:43.781 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995279,1725995281,1725995282,1725995284,1725995285,0 2024-09-10T19:08:43.781 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995279,1725995281,1725995282,1725995284,1725995285,0 2024-09-10T19:08:43.781 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995279,1725995281,1725995282,1725995284,1725995285,0 2024-09-10T19:08:43.781 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995279,1725995281,1725995282,1725995284,1725995285,1725995287,1725995288,0 2024-09-10T19:08:43.781 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995279,1725995281,1725995282,1725995284,1725995285,1725995287,1725995288,0 2024-09-10T19:08:43.781 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995279,1725995281,1725995282,1725995284,1725995285,1725995287,1725995288,0 2024-09-10T19:08:43.782 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995282,1725995284,1725995285,1725995287,1725995288,1725995290,1725995291,0 2024-09-10T19:08:43.782 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: first now 1725995282, trimmed 2024-09-10T19:08:43.782 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (19865 ms) 2024-09-10T19:08:43.782 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-10T19:08:43.782 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: foo0 2024-09-10T19:08:43.782 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-10T19:08:43.782 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14034 ms) 2024-09-10T19:08:43.782 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-10T19:08:44.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]': finished 2024-09-10T19:08:44.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi119-79499-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:44.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[29577]: osdmap e342: 8 total, 8 up, 8 in 2024-09-10T19:08:44.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi119-79499-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:44.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi119-79499-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:44.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:44.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:44.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:44.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:44.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:44.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]': finished 2024-09-10T19:08:44.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi119-79499-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:44.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[40674]: osdmap e342: 8 total, 8 up, 8 in 2024-09-10T19:08:44.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi119-79499-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:44.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi119-79499-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:44.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:44.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:44.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:44.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:44.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:44 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:44.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]': finished 2024-09-10T19:08:44.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi119-79499-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:44.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:44 smithi194 ceph-mon[32464]: osdmap e342: 8 total, 8 up, 8 in 2024-09-10T19:08:44.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi119-79499-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:44.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi119-79499-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:44.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:44.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:08:44.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:44.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-49"}]: dispatch 2024-09-10T19:08:44.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:44 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:45.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[29577]: pgmap v477: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:45.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[29577]: osdmap e343: 8 total, 8 up, 8 in 2024-09-10T19:08:45.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:45.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:45.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:08:45.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi119-79499-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi119-79499-65"}]': finished 2024-09-10T19:08:45.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[29577]: osdmap e344: 8 total, 8 up, 8 in 2024-09-10T19:08:45.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:45.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:45.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[40674]: pgmap v477: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:45.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[40674]: osdmap e343: 8 total, 8 up, 8 in 2024-09-10T19:08:45.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:45.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:45.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:08:45.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi119-79499-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi119-79499-65"}]': finished 2024-09-10T19:08:45.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[40674]: osdmap e344: 8 total, 8 up, 8 in 2024-09-10T19:08:45.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:45.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:45 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:45.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:45 smithi194 ceph-mon[32464]: pgmap v477: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:45.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:45 smithi194 ceph-mon[32464]: osdmap e343: 8 total, 8 up, 8 in 2024-09-10T19:08:45.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:45 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:45.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:45 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:45.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:45 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:08:45.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:45 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi119-79499-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi119-79499-65"}]': finished 2024-09-10T19:08:45.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:45 smithi194 ceph-mon[32464]: osdmap e344: 8 total, 8 up, 8 in 2024-09-10T19:08:45.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:45.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:45 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:46 smithi119 ceph-mon[40674]: pgmap v480: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:46 smithi119 ceph-mon[40674]: osdmap e345: 8 total, 8 up, 8 in 2024-09-10T19:08:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:46 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:46 smithi119 ceph-mon[29577]: pgmap v480: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:46.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:46.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:46 smithi119 ceph-mon[29577]: osdmap e345: 8 total, 8 up, 8 in 2024-09-10T19:08:46.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:46.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:46.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:46 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:47.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:46 smithi194 ceph-mon[32464]: pgmap v480: 236 pgs: 40 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:08:47.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:47.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:46 smithi194 ceph-mon[32464]: osdmap e345: 8 total, 8 up, 8 in 2024-09-10T19:08:47.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:47.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:08:47.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:46 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[29577]: osdmap e346: 8 total, 8 up, 8 in 2024-09-10T19:08:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:08:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:08:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:08:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[40674]: osdmap e346: 8 total, 8 up, 8 in 2024-09-10T19:08:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:08:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:08:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:08:47.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:48.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:47 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:08:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:47 smithi194 ceph-mon[32464]: osdmap e346: 8 total, 8 up, 8 in 2024-09-10T19:08:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:08:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:47 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:08:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:47 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:08:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:08:48.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:08:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:08:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:48 smithi194 ceph-mon[32464]: pgmap v483: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-51"}]': finished 2024-09-10T19:08:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]': finished 2024-09-10T19:08:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:48 smithi194 ceph-mon[32464]: osdmap e347: 8 total, 8 up, 8 in 2024-09-10T19:08:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-51", "mode": "writeback"}]: dispatch 2024-09-10T19:08:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-51", "mode": "writeback"}]: dispatch 2024-09-10T19:08:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:48 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:08:48.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[29577]: pgmap v483: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-51"}]': finished 2024-09-10T19:08:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]': finished 2024-09-10T19:08:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[29577]: osdmap e347: 8 total, 8 up, 8 in 2024-09-10T19:08:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-51", "mode": "writeback"}]: dispatch 2024-09-10T19:08:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-51", "mode": "writeback"}]: dispatch 2024-09-10T19:08:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:08:48.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[40674]: pgmap v483: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:08:48.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-51"}]': finished 2024-09-10T19:08:48.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi119-79499-65"}]': finished 2024-09-10T19:08:48.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[40674]: osdmap e347: 8 total, 8 up, 8 in 2024-09-10T19:08:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-51", "mode": "writeback"}]: dispatch 2024-09-10T19:08:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-51", "mode": "writeback"}]: dispatch 2024-09-10T19:08:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1915975214' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]: dispatch 2024-09-10T19:08:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:48 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:08:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-51", "mode": "writeback"}]': finished 2024-09-10T19:08:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]': finished 2024-09-10T19:08:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[29577]: osdmap e348: 8 total, 8 up, 8 in 2024-09-10T19:08:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi119-79499-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:49.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:49.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-51", "mode": "writeback"}]': finished 2024-09-10T19:08:49.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]': finished 2024-09-10T19:08:49.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[40674]: osdmap e348: 8 total, 8 up, 8 in 2024-09-10T19:08:49.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:49.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:49.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi119-79499-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:49.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:49.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:50.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:49 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:08:50.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-51", "mode": "writeback"}]': finished 2024-09-10T19:08:50.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi119-79499-65"}]': finished 2024-09-10T19:08:50.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:49 smithi194 ceph-mon[32464]: osdmap e348: 8 total, 8 up, 8 in 2024-09-10T19:08:50.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:50.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:50.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi119-79499-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:08:50.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:50.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:08:50.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:08:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:08:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:08:51.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:50 smithi194 ceph-mon[32464]: pgmap v486: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:08:51.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi119-79499-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:51.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:50 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:08:51.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:50 smithi194 ceph-mon[32464]: osdmap e349: 8 total, 8 up, 8 in 2024-09-10T19:08:51.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:51.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi119-79499-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:51.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:50 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:51.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:50 smithi119 ceph-mon[29577]: pgmap v486: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:08:51.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi119-79499-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:51.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:50 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:08:51.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:50 smithi119 ceph-mon[29577]: osdmap e349: 8 total, 8 up, 8 in 2024-09-10T19:08:51.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:51.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi119-79499-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:51.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:50 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:51.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:50 smithi119 ceph-mon[40674]: pgmap v486: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:08:51.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi119-79499-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:08:51.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:50 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:08:51.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:50 smithi119 ceph-mon[40674]: osdmap e349: 8 total, 8 up, 8 in 2024-09-10T19:08:51.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:51.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi119-79499-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:51.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:50 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:08:52.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:08:52.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:51 smithi194 ceph-mon[32464]: osdmap e350: 8 total, 8 up, 8 in 2024-09-10T19:08:52.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:52.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:52.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:51 smithi194 ceph-mon[32464]: pgmap v489: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:08:52.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:51 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:52.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi119-79499-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi119-79499-66"}]': finished 2024-09-10T19:08:52.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:08:52.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:51 smithi194 ceph-mon[32464]: osdmap e351: 8 total, 8 up, 8 in 2024-09-10T19:08:52.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:52.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:08:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[29577]: osdmap e350: 8 total, 8 up, 8 in 2024-09-10T19:08:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[29577]: pgmap v489: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:08:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi119-79499-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi119-79499-66"}]': finished 2024-09-10T19:08:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:08:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[29577]: osdmap e351: 8 total, 8 up, 8 in 2024-09-10T19:08:52.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:52.318 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:08:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[40674]: osdmap e350: 8 total, 8 up, 8 in 2024-09-10T19:08:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:08:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[40674]: pgmap v489: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:08:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:08:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi119-79499-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi119-79499-66"}]': finished 2024-09-10T19:08:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:08:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[40674]: osdmap e351: 8 total, 8 up, 8 in 2024-09-10T19:08:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:52.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:08:53.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:08:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:08:52] "GET /metrics HTTP/1.1" 200 228377 "" "Prometheus/2.43.0" 2024-09-10T19:08:53.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:52 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:53.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:08:53.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:52 smithi119 ceph-mon[29577]: osdmap e352: 8 total, 8 up, 8 in 2024-09-10T19:08:53.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:08:53.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:08:53.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:52 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:53.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:08:53.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:52 smithi119 ceph-mon[40674]: osdmap e352: 8 total, 8 up, 8 in 2024-09-10T19:08:53.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:08:53.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:08:53.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:52 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:08:53.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:08:53.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:52 smithi194 ceph-mon[32464]: osdmap e352: 8 total, 8 up, 8 in 2024-09-10T19:08:53.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:08:53.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:08:55.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:54 smithi194 ceph-mon[32464]: pgmap v492: 236 pgs: 1 creating+activating, 235 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 255 B/s wr, 4 op/s 2024-09-10T19:08:55.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:54 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "target_max_objects","val": "1"}]': finished 2024-09-10T19:08:55.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:54 smithi194 ceph-mon[32464]: osdmap e353: 8 total, 8 up, 8 in 2024-09-10T19:08:55.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:54 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:55.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:54 smithi119 ceph-mon[29577]: pgmap v492: 236 pgs: 1 creating+activating, 235 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 255 B/s wr, 4 op/s 2024-09-10T19:08:55.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:54 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "target_max_objects","val": "1"}]': finished 2024-09-10T19:08:55.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:54 smithi119 ceph-mon[29577]: osdmap e353: 8 total, 8 up, 8 in 2024-09-10T19:08:55.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:55.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:54 smithi119 ceph-mon[40674]: pgmap v492: 236 pgs: 1 creating+activating, 235 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 255 B/s wr, 4 op/s 2024-09-10T19:08:55.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:54 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-51","var": "target_max_objects","val": "1"}]': finished 2024-09-10T19:08:55.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:54 smithi119 ceph-mon[40674]: osdmap e353: 8 total, 8 up, 8 in 2024-09-10T19:08:55.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:54 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:55.601 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ P2 2024-09-10T19:08:55.602 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3012 ms) 2024-09-10T19:08:55.602 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-10T19:08:55.602 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7017 ms) 2024-09-10T19:08:55.602 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-10T19:08:55.602 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7028 ms) 2024-09-10T19:08:55.602 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-10T19:08:55.602 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7042 ms) 2024-09-10T19:08:55.602 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-10T19:08:55.602 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7023 ms) 2024-09-10T19:08:55.602 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-10T19:08:55.603 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7030 ms) 2024-09-10T19:08:55.603 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-10T19:08:55.603 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7030 ms) 2024-09-10T19:08:55.603 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139326 ms total) 2024-09-10T19:08:55.603 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: 2024-09-10T19:08:55.603 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-10T19:08:55.603 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (294347 ms total) 2024-09-10T19:08:55.603 INFO:tasks.workunit.client.0.smithi119.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-10T19:08:56.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi119-79499-66"}]': finished 2024-09-10T19:08:56.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:55 smithi194 ceph-mon[32464]: osdmap e354: 8 total, 8 up, 8 in 2024-09-10T19:08:56.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:56.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi119-79499-66"}]': finished 2024-09-10T19:08:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:55 smithi119 ceph-mon[29577]: osdmap e354: 8 total, 8 up, 8 in 2024-09-10T19:08:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:56.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi119-79499-66"}]': finished 2024-09-10T19:08:56.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:55 smithi119 ceph-mon[40674]: osdmap e354: 8 total, 8 up, 8 in 2024-09-10T19:08:56.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi119-79499-66"}]: dispatch 2024-09-10T19:08:57.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:56 smithi194 ceph-mon[32464]: pgmap v495: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-10T19:08:57.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:56 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-10T19:08:57.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi119-79499-66"}]': finished 2024-09-10T19:08:57.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:56 smithi194 ceph-mon[32464]: osdmap e355: 8 total, 8 up, 8 in 2024-09-10T19:08:57.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:56 smithi119 ceph-mon[29577]: pgmap v495: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-10T19:08:57.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:56 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-10T19:08:57.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi119-79499-66"}]': finished 2024-09-10T19:08:57.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:56 smithi119 ceph-mon[29577]: osdmap e355: 8 total, 8 up, 8 in 2024-09-10T19:08:57.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:56 smithi119 ceph-mon[40674]: pgmap v495: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-10T19:08:57.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:56 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-10T19:08:57.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1487426466' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi119-79499-66"}]': finished 2024-09-10T19:08:57.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:56 smithi119 ceph-mon[40674]: osdmap e355: 8 total, 8 up, 8 in 2024-09-10T19:08:58.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:58 smithi194 ceph-mon[32464]: pgmap v497: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-10T19:08:58.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:08:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:08:59.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:58 smithi119 ceph-mon[29577]: pgmap v497: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-10T19:08:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:58 smithi119 ceph-mon[40674]: pgmap v497: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-10T19:09:00.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:08:59 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:08:59 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:08:59 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:00.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:09:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:09:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:09:01.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:00 smithi194 ceph-mon[32464]: pgmap v498: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 0 op/s 2024-09-10T19:09:01.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:00 smithi119 ceph-mon[29577]: pgmap v498: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 0 op/s 2024-09-10T19:09:01.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:00 smithi119 ceph-mon[40674]: pgmap v498: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 682 B/s rd, 0 B/s wr, 0 op/s 2024-09-10T19:09:03.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:02 smithi194 ceph-mon[32464]: pgmap v499: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 676 B/s rd, 0 B/s wr, 0 op/s 2024-09-10T19:09:03.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:09:03.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:03.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:09:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:09:02] "GET /metrics HTTP/1.1" 200 228503 "" "Prometheus/2.43.0" 2024-09-10T19:09:03.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:02 smithi119 ceph-mon[29577]: pgmap v499: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 676 B/s rd, 0 B/s wr, 0 op/s 2024-09-10T19:09:03.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:09:03.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:03.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:02 smithi119 ceph-mon[40674]: pgmap v499: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 676 B/s rd, 0 B/s wr, 0 op/s 2024-09-10T19:09:03.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:09:03.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:04.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:04.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:04.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:04.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:04.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:04.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:05.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:04 smithi194 ceph-mon[32464]: pgmap v500: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:09:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:04 smithi194 ceph-mon[32464]: osdmap e356: 8 total, 8 up, 8 in 2024-09-10T19:09:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:09:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:09:05.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:04 smithi119 ceph-mon[29577]: pgmap v500: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:09:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:04 smithi119 ceph-mon[29577]: osdmap e356: 8 total, 8 up, 8 in 2024-09-10T19:09:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:09:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:09:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:04 smithi119 ceph-mon[40674]: pgmap v500: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:09:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:04 smithi119 ceph-mon[40674]: osdmap e356: 8 total, 8 up, 8 in 2024-09-10T19:09:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:09:05.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:09:06.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]': finished 2024-09-10T19:09:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:05 smithi194 ceph-mon[32464]: osdmap e357: 8 total, 8 up, 8 in 2024-09-10T19:09:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:09:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:09:06.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]': finished 2024-09-10T19:09:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:05 smithi119 ceph-mon[29577]: osdmap e357: 8 total, 8 up, 8 in 2024-09-10T19:09:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:09:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:09:06.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]': finished 2024-09-10T19:09:06.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:05 smithi119 ceph-mon[40674]: osdmap e357: 8 total, 8 up, 8 in 2024-09-10T19:09:06.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:06.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:06.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:09:06.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-51"}]: dispatch 2024-09-10T19:09:07.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:06 smithi194 ceph-mon[32464]: pgmap v503: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:06 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-10T19:09:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:06 smithi194 ceph-mon[32464]: osdmap e358: 8 total, 8 up, 8 in 2024-09-10T19:09:07.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:06 smithi119 ceph-mon[29577]: pgmap v503: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:07.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:06 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-10T19:09:07.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:06 smithi119 ceph-mon[29577]: osdmap e358: 8 total, 8 up, 8 in 2024-09-10T19:09:07.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:06 smithi119 ceph-mon[40674]: pgmap v503: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:07.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:06 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-10T19:09:07.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:06 smithi119 ceph-mon[40674]: osdmap e358: 8 total, 8 up, 8 in 2024-09-10T19:09:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:07 smithi194 ceph-mon[32464]: osdmap e359: 8 total, 8 up, 8 in 2024-09-10T19:09:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:07 smithi194 ceph-mon[32464]: pgmap v506: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:09:08.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:07 smithi119 ceph-mon[29577]: osdmap e359: 8 total, 8 up, 8 in 2024-09-10T19:09:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:07 smithi119 ceph-mon[29577]: pgmap v506: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:09:08.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:07 smithi119 ceph-mon[40674]: osdmap e359: 8 total, 8 up, 8 in 2024-09-10T19:09:08.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:08.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:08.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:07 smithi119 ceph-mon[40674]: pgmap v506: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:09:08.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:09:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:09:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:08 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:09:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:08 smithi194 ceph-mon[32464]: osdmap e360: 8 total, 8 up, 8 in 2024-09-10T19:09:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-53"}]: dispatch 2024-09-10T19:09:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-53"}]: dispatch 2024-09-10T19:09:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:08 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:09.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:09:09.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:09.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[29577]: osdmap e360: 8 total, 8 up, 8 in 2024-09-10T19:09:09.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:09.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:09.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-53"}]: dispatch 2024-09-10T19:09:09.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-53"}]: dispatch 2024-09-10T19:09:09.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:09.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:09:09.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:09.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[40674]: osdmap e360: 8 total, 8 up, 8 in 2024-09-10T19:09:09.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:09.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:09.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-53"}]: dispatch 2024-09-10T19:09:09.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-53"}]: dispatch 2024-09-10T19:09:09.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:08 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:10.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:09 smithi194 ceph-mon[32464]: osdmap e361: 8 total, 8 up, 8 in 2024-09-10T19:09:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:09 smithi194 ceph-mon[32464]: pgmap v509: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-10T19:09:10.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:09 smithi119 ceph-mon[40674]: osdmap e361: 8 total, 8 up, 8 in 2024-09-10T19:09:10.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:09 smithi119 ceph-mon[40674]: pgmap v509: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-10T19:09:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:09 smithi119 ceph-mon[29577]: osdmap e361: 8 total, 8 up, 8 in 2024-09-10T19:09:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:09 smithi119 ceph-mon[29577]: pgmap v509: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-10T19:09:10.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:09:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:09:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:09:11.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:10 smithi194 ceph-mon[32464]: osdmap e362: 8 total, 8 up, 8 in 2024-09-10T19:09:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:11.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:10 smithi119 ceph-mon[40674]: osdmap e362: 8 total, 8 up, 8 in 2024-09-10T19:09:11.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:11.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:11.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:10 smithi119 ceph-mon[29577]: osdmap e362: 8 total, 8 up, 8 in 2024-09-10T19:09:11.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:11.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:12.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:12.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:11 smithi194 ceph-mon[32464]: osdmap e363: 8 total, 8 up, 8 in 2024-09-10T19:09:12.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:09:12.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:12.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:12.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-55"}]: dispatch 2024-09-10T19:09:12.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-55"}]: dispatch 2024-09-10T19:09:12.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:11 smithi194 ceph-mon[32464]: pgmap v512: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:09:12.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:12.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[29577]: osdmap e363: 8 total, 8 up, 8 in 2024-09-10T19:09:12.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:09:12.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:12.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:12.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-55"}]: dispatch 2024-09-10T19:09:12.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-55"}]: dispatch 2024-09-10T19:09:12.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[29577]: pgmap v512: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:09:12.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:12.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[40674]: osdmap e363: 8 total, 8 up, 8 in 2024-09-10T19:09:12.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:09:12.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:12.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:12.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-55"}]: dispatch 2024-09-10T19:09:12.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-55"}]: dispatch 2024-09-10T19:09:12.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:11 smithi119 ceph-mon[40674]: pgmap v512: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:09:13.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:12 smithi194 ceph-mon[32464]: osdmap e364: 8 total, 8 up, 8 in 2024-09-10T19:09:13.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:12 smithi119 ceph-mon[29577]: osdmap e364: 8 total, 8 up, 8 in 2024-09-10T19:09:13.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:09:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:09:12] "GET /metrics HTTP/1.1" 200 228503 "" "Prometheus/2.43.0" 2024-09-10T19:09:13.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:12 smithi119 ceph-mon[40674]: osdmap e364: 8 total, 8 up, 8 in 2024-09-10T19:09:14.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:13 smithi194 ceph-mon[32464]: osdmap e365: 8 total, 8 up, 8 in 2024-09-10T19:09:14.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:14.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:14.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:13 smithi194 ceph-mon[32464]: pgmap v515: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:09:14.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:13 smithi119 ceph-mon[29577]: osdmap e365: 8 total, 8 up, 8 in 2024-09-10T19:09:14.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:14.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:14.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:13 smithi119 ceph-mon[29577]: pgmap v515: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:09:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:13 smithi119 ceph-mon[40674]: osdmap e365: 8 total, 8 up, 8 in 2024-09-10T19:09:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:13 smithi119 ceph-mon[40674]: pgmap v515: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:09:15.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:14 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:09:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:14 smithi194 ceph-mon[32464]: osdmap e366: 8 total, 8 up, 8 in 2024-09-10T19:09:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-57"}]: dispatch 2024-09-10T19:09:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-57"}]: dispatch 2024-09-10T19:09:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:14 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:09:15.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:15.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:14 smithi119 ceph-mon[40674]: osdmap e366: 8 total, 8 up, 8 in 2024-09-10T19:09:15.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:15.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:15.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-57"}]: dispatch 2024-09-10T19:09:15.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-57"}]: dispatch 2024-09-10T19:09:15.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:14 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:09:15.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:15.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:14 smithi119 ceph-mon[29577]: osdmap e366: 8 total, 8 up, 8 in 2024-09-10T19:09:15.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:15.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:15.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-57"}]: dispatch 2024-09-10T19:09:15.069 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-57"}]: dispatch 2024-09-10T19:09:16.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:15 smithi194 ceph-mon[32464]: osdmap e367: 8 total, 8 up, 8 in 2024-09-10T19:09:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:15 smithi194 ceph-mon[32464]: pgmap v518: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:09:16.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:15 smithi119 ceph-mon[29577]: osdmap e367: 8 total, 8 up, 8 in 2024-09-10T19:09:16.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:15 smithi119 ceph-mon[29577]: pgmap v518: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:09:16.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:15 smithi119 ceph-mon[40674]: osdmap e367: 8 total, 8 up, 8 in 2024-09-10T19:09:16.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:15 smithi119 ceph-mon[40674]: pgmap v518: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:09:17.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:16 smithi194 ceph-mon[32464]: osdmap e368: 8 total, 8 up, 8 in 2024-09-10T19:09:17.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:17.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:17.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:16 smithi119 ceph-mon[29577]: osdmap e368: 8 total, 8 up, 8 in 2024-09-10T19:09:17.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:17.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:17.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:16 smithi119 ceph-mon[40674]: osdmap e368: 8 total, 8 up, 8 in 2024-09-10T19:09:17.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:17.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:18.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:17 smithi194 ceph-mon[32464]: osdmap e369: 8 total, 8 up, 8 in 2024-09-10T19:09:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:09:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-59"}]: dispatch 2024-09-10T19:09:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-59"}]: dispatch 2024-09-10T19:09:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:09:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:17 smithi194 ceph-mon[32464]: pgmap v521: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:09:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[29577]: osdmap e369: 8 total, 8 up, 8 in 2024-09-10T19:09:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:09:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-59"}]: dispatch 2024-09-10T19:09:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-59"}]: dispatch 2024-09-10T19:09:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:09:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[29577]: pgmap v521: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:09:18.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:18.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[40674]: osdmap e369: 8 total, 8 up, 8 in 2024-09-10T19:09:18.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:09:18.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:18.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:18.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-59"}]: dispatch 2024-09-10T19:09:18.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-59"}]: dispatch 2024-09-10T19:09:18.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:09:18.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:18.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:17 smithi119 ceph-mon[40674]: pgmap v521: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:09:18.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:09:19.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: osdmap e370: 8 total, 8 up, 8 in 2024-09-10T19:09:19.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.7", "id": [5, 4]}]: dispatch 2024-09-10T19:09:19.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.9", "id": [7, 2]}]: dispatch 2024-09-10T19:09:19.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.c", "id": [7, 6]}]: dispatch 2024-09-10T19:09:19.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1f", "id": [7, 2]}]: dispatch 2024-09-10T19:09:19.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]: dispatch 2024-09-10T19:09:19.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.f", "id": [1, 7]}]: dispatch 2024-09-10T19:09:19.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:19.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.7", "id": [5, 4]}]': finished 2024-09-10T19:09:19.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.9", "id": [7, 2]}]': finished 2024-09-10T19:09:19.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.c", "id": [7, 6]}]': finished 2024-09-10T19:09:19.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1f", "id": [7, 2]}]': finished 2024-09-10T19:09:19.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]': finished 2024-09-10T19:09:19.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.f", "id": [1, 7]}]': finished 2024-09-10T19:09:19.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: osdmap e371: 8 total, 8 up, 8 in 2024-09-10T19:09:19.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:19.219 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:19.219 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: osdmap e370: 8 total, 8 up, 8 in 2024-09-10T19:09:19.220 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.7", "id": [5, 4]}]: dispatch 2024-09-10T19:09:19.220 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.9", "id": [7, 2]}]: dispatch 2024-09-10T19:09:19.220 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.c", "id": [7, 6]}]: dispatch 2024-09-10T19:09:19.220 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1f", "id": [7, 2]}]: dispatch 2024-09-10T19:09:19.220 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]: dispatch 2024-09-10T19:09:19.220 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.f", "id": [1, 7]}]: dispatch 2024-09-10T19:09:19.220 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:19.220 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.7", "id": [5, 4]}]': finished 2024-09-10T19:09:19.220 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.9", "id": [7, 2]}]': finished 2024-09-10T19:09:19.220 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.c", "id": [7, 6]}]': finished 2024-09-10T19:09:19.220 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1f", "id": [7, 2]}]': finished 2024-09-10T19:09:19.220 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]': finished 2024-09-10T19:09:19.220 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.f", "id": [1, 7]}]': finished 2024-09-10T19:09:19.221 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: osdmap e371: 8 total, 8 up, 8 in 2024-09-10T19:09:19.221 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:19.221 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: osdmap e370: 8 total, 8 up, 8 in 2024-09-10T19:09:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.7", "id": [5, 4]}]: dispatch 2024-09-10T19:09:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.9", "id": [7, 2]}]: dispatch 2024-09-10T19:09:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.c", "id": [7, 6]}]: dispatch 2024-09-10T19:09:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1f", "id": [7, 2]}]: dispatch 2024-09-10T19:09:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]: dispatch 2024-09-10T19:09:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.f", "id": [1, 7]}]: dispatch 2024-09-10T19:09:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.7", "id": [5, 4]}]': finished 2024-09-10T19:09:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.9", "id": [7, 2]}]': finished 2024-09-10T19:09:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.c", "id": [7, 6]}]': finished 2024-09-10T19:09:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "9.1f", "id": [7, 2]}]': finished 2024-09-10T19:09:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.1", "id": [1, 3]}]': finished 2024-09-10T19:09:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "63.f", "id": [1, 7]}]': finished 2024-09-10T19:09:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: osdmap e371: 8 total, 8 up, 8 in 2024-09-10T19:09:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:19.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:20.269 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:09:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:09:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:09:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:19 smithi194 ceph-mon[32464]: pgmap v524: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:09:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:19 smithi194 ceph-mon[32464]: osdmap e372: 8 total, 8 up, 8 in 2024-09-10T19:09:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:09:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:09:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:09:20.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:19 smithi119 ceph-mon[29577]: pgmap v524: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:09:20.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:20.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:19 smithi119 ceph-mon[29577]: osdmap e372: 8 total, 8 up, 8 in 2024-09-10T19:09:20.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:09:20.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:09:20.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:09:20.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:19 smithi119 ceph-mon[40674]: pgmap v524: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:09:20.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:20.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:19 smithi119 ceph-mon[40674]: osdmap e372: 8 total, 8 up, 8 in 2024-09-10T19:09:20.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:09:20.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:09:20.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:09:21.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:20 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:09:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:09:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:20 smithi194 ceph-mon[32464]: osdmap e373: 8 total, 8 up, 8 in 2024-09-10T19:09:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-61"}]: dispatch 2024-09-10T19:09:21.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:20 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-61"}]: dispatch 2024-09-10T19:09:21.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:20 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:09:21.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:09:21.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:20 smithi119 ceph-mon[29577]: osdmap e373: 8 total, 8 up, 8 in 2024-09-10T19:09:21.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:21.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:21.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-61"}]: dispatch 2024-09-10T19:09:21.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:20 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-61"}]: dispatch 2024-09-10T19:09:21.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:20 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:09:21.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:09:21.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:20 smithi119 ceph-mon[40674]: osdmap e373: 8 total, 8 up, 8 in 2024-09-10T19:09:21.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:21.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:09:21.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-61"}]: dispatch 2024-09-10T19:09:21.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:20 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-61"}]: dispatch 2024-09-10T19:09:22.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:21 smithi194 ceph-mon[32464]: pgmap v527: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:09:22.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:21 smithi194 ceph-mon[32464]: osdmap e374: 8 total, 8 up, 8 in 2024-09-10T19:09:22.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:21 smithi119 ceph-mon[29577]: pgmap v527: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:09:22.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:21 smithi119 ceph-mon[29577]: osdmap e374: 8 total, 8 up, 8 in 2024-09-10T19:09:22.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:21 smithi119 ceph-mon[40674]: pgmap v527: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-10T19:09:22.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:21 smithi119 ceph-mon[40674]: osdmap e374: 8 total, 8 up, 8 in 2024-09-10T19:09:23.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:09:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:09:22] "GET /metrics HTTP/1.1" 200 228465 "" "Prometheus/2.43.0" 2024-09-10T19:09:24.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:23 smithi194 ceph-mon[32464]: osdmap e375: 8 total, 8 up, 8 in 2024-09-10T19:09:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:23 smithi194 ceph-mon[32464]: pgmap v530: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 30 B/s, 10 objects/s recovering 2024-09-10T19:09:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:23 smithi119 ceph-mon[29577]: osdmap e375: 8 total, 8 up, 8 in 2024-09-10T19:09:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:23 smithi119 ceph-mon[29577]: pgmap v530: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 30 B/s, 10 objects/s recovering 2024-09-10T19:09:24.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:23 smithi119 ceph-mon[40674]: osdmap e375: 8 total, 8 up, 8 in 2024-09-10T19:09:24.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:24.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:09:24.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:23 smithi119 ceph-mon[40674]: pgmap v530: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 30 B/s, 10 objects/s recovering 2024-09-10T19:09:25.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:25.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:24 smithi194 ceph-mon[32464]: osdmap e376: 8 total, 8 up, 8 in 2024-09-10T19:09:25.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:09:25.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:09:25.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:09:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:24 smithi119 ceph-mon[29577]: osdmap e376: 8 total, 8 up, 8 in 2024-09-10T19:09:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:09:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:09:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:09:25.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:09:25.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:24 smithi119 ceph-mon[40674]: osdmap e376: 8 total, 8 up, 8 in 2024-09-10T19:09:25.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:09:25.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:09:25.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:09:26.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:09:26.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:25 smithi194 ceph-mon[32464]: osdmap e377: 8 total, 8 up, 8 in 2024-09-10T19:09:26.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:25 smithi194 ceph-mon[32464]: pgmap v533: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 30 B/s, 10 objects/s recovering 2024-09-10T19:09:26.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:09:26.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:25 smithi119 ceph-mon[29577]: osdmap e377: 8 total, 8 up, 8 in 2024-09-10T19:09:26.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:25 smithi119 ceph-mon[29577]: pgmap v533: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 30 B/s, 10 objects/s recovering 2024-09-10T19:09:26.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:09:26.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:25 smithi119 ceph-mon[40674]: osdmap e377: 8 total, 8 up, 8 in 2024-09-10T19:09:26.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:25 smithi119 ceph-mon[40674]: pgmap v533: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s; 30 B/s, 10 objects/s recovering 2024-09-10T19:09:27.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:26 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:09:27.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:26 smithi194 ceph-mon[32464]: osdmap e378: 8 total, 8 up, 8 in 2024-09-10T19:09:27.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:26 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:09:27.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:26 smithi119 ceph-mon[29577]: osdmap e378: 8 total, 8 up, 8 in 2024-09-10T19:09:27.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:26 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:09:27.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:26 smithi119 ceph-mon[40674]: osdmap e378: 8 total, 8 up, 8 in 2024-09-10T19:09:28.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:27 smithi194 ceph-mon[32464]: osdmap e379: 8 total, 8 up, 8 in 2024-09-10T19:09:28.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:27 smithi194 ceph-mon[32464]: pgmap v536: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:09:28.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:27 smithi119 ceph-mon[29577]: osdmap e379: 8 total, 8 up, 8 in 2024-09-10T19:09:28.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:27 smithi119 ceph-mon[29577]: pgmap v536: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:09:28.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:27 smithi119 ceph-mon[40674]: osdmap e379: 8 total, 8 up, 8 in 2024-09-10T19:09:28.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:27 smithi119 ceph-mon[40674]: pgmap v536: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:09:28.765 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:09:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:09:29.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:28 smithi194 ceph-mon[32464]: osdmap e380: 8 total, 8 up, 8 in 2024-09-10T19:09:29.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:28 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:29.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:28 smithi119 ceph-mon[29577]: osdmap e380: 8 total, 8 up, 8 in 2024-09-10T19:09:29.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:28 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:29.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:28 smithi119 ceph-mon[40674]: osdmap e380: 8 total, 8 up, 8 in 2024-09-10T19:09:29.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:28 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:30.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:29 smithi119 ceph-mon[40674]: osdmap e381: 8 total, 8 up, 8 in 2024-09-10T19:09:30.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:29 smithi119 ceph-mon[40674]: pgmap v539: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-09-10T19:09:30.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:29 smithi119 ceph-mon[29577]: osdmap e381: 8 total, 8 up, 8 in 2024-09-10T19:09:30.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:29 smithi119 ceph-mon[29577]: pgmap v539: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-09-10T19:09:30.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:29 smithi194 ceph-mon[32464]: osdmap e381: 8 total, 8 up, 8 in 2024-09-10T19:09:30.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:29 smithi194 ceph-mon[32464]: pgmap v539: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2024-09-10T19:09:30.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:09:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:09:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:09:32.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:32 smithi194 ceph-mon[32464]: pgmap v540: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-09-10T19:09:32.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:32.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:32 smithi119 ceph-mon[29577]: pgmap v540: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-09-10T19:09:32.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:32.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:32 smithi119 ceph-mon[40674]: pgmap v540: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.0 KiB/s wr, 5 op/s 2024-09-10T19:09:32.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:33.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:09:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:09:32] "GET /metrics HTTP/1.1" 200 228506 "" "Prometheus/2.43.0" 2024-09-10T19:09:34.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:34 smithi194 ceph-mon[32464]: pgmap v541: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-09-10T19:09:34.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:34 smithi119 ceph-mon[29577]: pgmap v541: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-09-10T19:09:34.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:34 smithi119 ceph-mon[40674]: pgmap v541: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-09-10T19:09:36.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:36 smithi194 ceph-mon[32464]: pgmap v542: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-09-10T19:09:36.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:36 smithi119 ceph-mon[29577]: pgmap v542: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-09-10T19:09:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:36 smithi119 ceph-mon[40674]: pgmap v542: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.1 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-09-10T19:09:38.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:09:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:09:38.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:38 smithi194 ceph-mon[32464]: pgmap v543: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-10T19:09:38.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:38 smithi194 ceph-mon[32464]: osdmap e382: 8 total, 8 up, 8 in 2024-09-10T19:09:38.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:38 smithi119 ceph-mon[29577]: pgmap v543: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-10T19:09:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:38 smithi119 ceph-mon[29577]: osdmap e382: 8 total, 8 up, 8 in 2024-09-10T19:09:38.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:38 smithi119 ceph-mon[40674]: pgmap v543: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-09-10T19:09:38.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:38 smithi119 ceph-mon[40674]: osdmap e382: 8 total, 8 up, 8 in 2024-09-10T19:09:39.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:39 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:39 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:39.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:39 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:40.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:40 smithi194 ceph-mon[32464]: pgmap v545: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:40.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:40 smithi194 ceph-mon[32464]: osdmap e383: 8 total, 8 up, 8 in 2024-09-10T19:09:40.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:09:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:09:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:09:40.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:40 smithi119 ceph-mon[29577]: pgmap v545: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:40.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:40 smithi119 ceph-mon[29577]: osdmap e383: 8 total, 8 up, 8 in 2024-09-10T19:09:40.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:40 smithi119 ceph-mon[40674]: pgmap v545: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:40.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:40 smithi119 ceph-mon[40674]: osdmap e383: 8 total, 8 up, 8 in 2024-09-10T19:09:42.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:42 smithi194 ceph-mon[32464]: pgmap v547: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:42 smithi119 ceph-mon[29577]: pgmap v547: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:42 smithi119 ceph-mon[40674]: pgmap v547: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:43.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:09:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:09:42] "GET /metrics HTTP/1.1" 200 228506 "" "Prometheus/2.43.0" 2024-09-10T19:09:44.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:44 smithi194 ceph-mon[32464]: pgmap v548: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:44.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:44 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:44.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:44 smithi119 ceph-mon[29577]: pgmap v548: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:44.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:44 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:44.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:44 smithi119 ceph-mon[40674]: pgmap v548: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:44.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:44 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:45.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:45 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:45.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:45 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:45.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:45 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:09:45.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:45 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:45.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:45 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:45.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:45 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:09:45.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:45 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:45.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:45 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:45.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:45 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:09:46.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:46 smithi194 ceph-mon[32464]: pgmap v549: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:46 smithi119 ceph-mon[29577]: pgmap v549: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:46 smithi119 ceph-mon[40674]: pgmap v549: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:47.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:47.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:47 smithi194 ceph-mon[32464]: osdmap e384: 8 total, 8 up, 8 in 2024-09-10T19:09:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:47 smithi119 ceph-mon[29577]: osdmap e384: 8 total, 8 up, 8 in 2024-09-10T19:09:47.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:09:47.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:47 smithi119 ceph-mon[40674]: osdmap e384: 8 total, 8 up, 8 in 2024-09-10T19:09:48.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:48 smithi194 ceph-mon[32464]: pgmap v550: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:48.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:09:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:09:48.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:48 smithi119 ceph-mon[29577]: pgmap v550: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:48.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:48 smithi119 ceph-mon[40674]: pgmap v550: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:49.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:49 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:49.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:49 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:49.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:49 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:50.507 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:09:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:09:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:09:50.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:50 smithi194 ceph-mon[32464]: pgmap v552: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:50.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:50 smithi194 ceph-mon[32464]: osdmap e385: 8 total, 8 up, 8 in 2024-09-10T19:09:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:50 smithi119 ceph-mon[29577]: pgmap v552: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:50 smithi119 ceph-mon[29577]: osdmap e385: 8 total, 8 up, 8 in 2024-09-10T19:09:50.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:50 smithi119 ceph-mon[40674]: pgmap v552: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:09:50.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:50 smithi119 ceph-mon[40674]: osdmap e385: 8 total, 8 up, 8 in 2024-09-10T19:09:52.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:52 smithi194 ceph-mon[32464]: pgmap v554: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:52 smithi119 ceph-mon[29577]: pgmap v554: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:52.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:09:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:09:52] "GET /metrics HTTP/1.1" 200 228487 "" "Prometheus/2.43.0" 2024-09-10T19:09:52.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:52 smithi119 ceph-mon[40674]: pgmap v554: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:09:54.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:54 smithi194 ceph-mon[32464]: pgmap v555: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:09:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:54 smithi119 ceph-mon[29577]: pgmap v555: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:09:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:54 smithi119 ceph-mon[40674]: pgmap v555: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:09:56.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:56 smithi194 ceph-mon[32464]: pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:09:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:56 smithi119 ceph-mon[29577]: pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:09:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:56 smithi119 ceph-mon[40674]: pgmap v556: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:09:58.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:58 smithi194 ceph-mon[32464]: pgmap v557: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:09:58.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:58 smithi194 ceph-mon[32464]: osdmap e386: 8 total, 8 up, 8 in 2024-09-10T19:09:58.519 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:09:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:09:58.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:58 smithi119 ceph-mon[29577]: pgmap v557: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:09:58.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:58 smithi119 ceph-mon[29577]: osdmap e386: 8 total, 8 up, 8 in 2024-09-10T19:09:58.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:58 smithi119 ceph-mon[40674]: pgmap v557: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:09:58.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:58 smithi119 ceph-mon[40674]: osdmap e386: 8 total, 8 up, 8 in 2024-09-10T19:09:59.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:09:59 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:59.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:09:59 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:09:59.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:09:59 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:00.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:10:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:10:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:10:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:00 smithi194 ceph-mon[32464]: pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:10:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:00 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-63"}]: dispatch 2024-09-10T19:10:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:00 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-63"}]: dispatch 2024-09-10T19:10:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:00 smithi194 ceph-mon[32464]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-10T19:10:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:00 smithi194 ceph-mon[32464]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-10T19:10:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:00 smithi194 ceph-mon[32464]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-09-10T19:10:00.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:00 smithi194 ceph-mon[32464]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-10T19:10:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[29577]: pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:10:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-63"}]: dispatch 2024-09-10T19:10:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-63"}]: dispatch 2024-09-10T19:10:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[29577]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-10T19:10:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[29577]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-10T19:10:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[29577]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-09-10T19:10:00.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[29577]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-10T19:10:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[40674]: pgmap v559: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:10:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-63"}]: dispatch 2024-09-10T19:10:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-63"}]: dispatch 2024-09-10T19:10:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[40674]: Health detail: HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-10T19:10:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[40674]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-10T19:10:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[40674]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-09-10T19:10:00.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:00 smithi119 ceph-mon[40674]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-10T19:10:01.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:01 smithi194 ceph-mon[32464]: osdmap e387: 8 total, 8 up, 8 in 2024-09-10T19:10:01.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:01 smithi119 ceph-mon[29577]: osdmap e387: 8 total, 8 up, 8 in 2024-09-10T19:10:01.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:01 smithi119 ceph-mon[40674]: osdmap e387: 8 total, 8 up, 8 in 2024-09-10T19:10:02.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:02 smithi119 ceph-mon[29577]: pgmap v561: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:02 smithi119 ceph-mon[29577]: osdmap e388: 8 total, 8 up, 8 in 2024-09-10T19:10:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:02 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:02.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:02.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:02 smithi119 ceph-mon[40674]: pgmap v561: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:02.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:02 smithi119 ceph-mon[40674]: osdmap e388: 8 total, 8 up, 8 in 2024-09-10T19:10:02.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:02.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:02 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:02.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:02.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:02 smithi194 ceph-mon[32464]: pgmap v561: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:02 smithi194 ceph-mon[32464]: osdmap e388: 8 total, 8 up, 8 in 2024-09-10T19:10:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:02 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:03.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:10:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:10:02] "GET /metrics HTTP/1.1" 200 228408 "" "Prometheus/2.43.0" 2024-09-10T19:10:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:03 smithi119 ceph-mon[40674]: osdmap e389: 8 total, 8 up, 8 in 2024-09-10T19:10:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:10:03.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:10:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:03 smithi119 ceph-mon[29577]: osdmap e389: 8 total, 8 up, 8 in 2024-09-10T19:10:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:10:03.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:10:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:03 smithi194 ceph-mon[32464]: osdmap e389: 8 total, 8 up, 8 in 2024-09-10T19:10:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:10:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:10:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:04 smithi119 ceph-mon[29577]: pgmap v564: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:10:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:04 smithi119 ceph-mon[29577]: osdmap e390: 8 total, 8 up, 8 in 2024-09-10T19:10:04.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:04 smithi119 ceph-mon[40674]: pgmap v564: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:04.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:10:04.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:04 smithi119 ceph-mon[40674]: osdmap e390: 8 total, 8 up, 8 in 2024-09-10T19:10:04.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:04 smithi194 ceph-mon[32464]: pgmap v564: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:04.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:10:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:04 smithi194 ceph-mon[32464]: osdmap e390: 8 total, 8 up, 8 in 2024-09-10T19:10:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:05 smithi119 ceph-mon[29577]: osdmap e391: 8 total, 8 up, 8 in 2024-09-10T19:10:05.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:05 smithi119 ceph-mon[40674]: osdmap e391: 8 total, 8 up, 8 in 2024-09-10T19:10:05.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:05 smithi194 ceph-mon[32464]: osdmap e391: 8 total, 8 up, 8 in 2024-09-10T19:10:06.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:06 smithi119 ceph-mon[29577]: pgmap v567: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:06.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:06 smithi119 ceph-mon[29577]: osdmap e392: 8 total, 8 up, 8 in 2024-09-10T19:10:06.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:06 smithi119 ceph-mon[40674]: pgmap v567: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:06.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:06 smithi119 ceph-mon[40674]: osdmap e392: 8 total, 8 up, 8 in 2024-09-10T19:10:06.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:06 smithi194 ceph-mon[32464]: pgmap v567: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:06.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:06 smithi194 ceph-mon[32464]: osdmap e392: 8 total, 8 up, 8 in 2024-09-10T19:10:07.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:07 smithi119 ceph-mon[29577]: osdmap e393: 8 total, 8 up, 8 in 2024-09-10T19:10:07.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:07 smithi119 ceph-mon[40674]: osdmap e393: 8 total, 8 up, 8 in 2024-09-10T19:10:07.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:07 smithi194 ceph-mon[32464]: osdmap e393: 8 total, 8 up, 8 in 2024-09-10T19:10:08.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:08 smithi119 ceph-mon[29577]: pgmap v570: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:08.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:08 smithi119 ceph-mon[40674]: pgmap v570: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:08.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:08 smithi194 ceph-mon[32464]: pgmap v570: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:08.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:10:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:10:09.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:09 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:09.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:09 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:09.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:09 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:10.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:10 smithi194 ceph-mon[32464]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-09-10T19:10:10.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:10:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:10:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:10:10.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:10 smithi119 ceph-mon[29577]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-09-10T19:10:10.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:10 smithi119 ceph-mon[40674]: pgmap v571: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-09-10T19:10:12.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:12 smithi119 ceph-mon[29577]: pgmap v572: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-10T19:10:12.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:12 smithi119 ceph-mon[29577]: osdmap e394: 8 total, 8 up, 8 in 2024-09-10T19:10:12.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:12 smithi119 ceph-mon[40674]: pgmap v572: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-10T19:10:12.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:12 smithi119 ceph-mon[40674]: osdmap e394: 8 total, 8 up, 8 in 2024-09-10T19:10:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:12 smithi194 ceph-mon[32464]: pgmap v572: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-10T19:10:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:12 smithi194 ceph-mon[32464]: osdmap e394: 8 total, 8 up, 8 in 2024-09-10T19:10:13.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:10:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:10:12] "GET /metrics HTTP/1.1" 200 228408 "" "Prometheus/2.43.0" 2024-09-10T19:10:14.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:14 smithi194 ceph-mon[32464]: pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-09-10T19:10:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:14 smithi119 ceph-mon[29577]: pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-09-10T19:10:14.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:14 smithi119 ceph-mon[40674]: pgmap v574: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-09-10T19:10:16.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:16 smithi194 ceph-mon[32464]: pgmap v575: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-09-10T19:10:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-65"}]: dispatch 2024-09-10T19:10:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-65"}]: dispatch 2024-09-10T19:10:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:16 smithi119 ceph-mon[29577]: pgmap v575: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-09-10T19:10:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-65"}]: dispatch 2024-09-10T19:10:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-65"}]: dispatch 2024-09-10T19:10:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:16 smithi119 ceph-mon[40674]: pgmap v575: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-09-10T19:10:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:16.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-65"}]: dispatch 2024-09-10T19:10:16.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-65"}]: dispatch 2024-09-10T19:10:17.712 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:17.712 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:17.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:18.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:18 smithi194 ceph-mon[32464]: pgmap v576: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-10T19:10:18.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:18 smithi194 ceph-mon[32464]: osdmap e395: 8 total, 8 up, 8 in 2024-09-10T19:10:18.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:10:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:10:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:18 smithi119 ceph-mon[29577]: pgmap v576: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-10T19:10:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:18 smithi119 ceph-mon[29577]: osdmap e395: 8 total, 8 up, 8 in 2024-09-10T19:10:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:18 smithi119 ceph-mon[40674]: pgmap v576: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2024-09-10T19:10:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:18 smithi119 ceph-mon[40674]: osdmap e395: 8 total, 8 up, 8 in 2024-09-10T19:10:19.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:19 smithi119 ceph-mon[29577]: osdmap e396: 8 total, 8 up, 8 in 2024-09-10T19:10:19.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:19.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:19.719 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:19 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:19.719 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:19.719 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:19 smithi119 ceph-mon[29577]: osdmap e397: 8 total, 8 up, 8 in 2024-09-10T19:10:19.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:19 smithi119 ceph-mon[40674]: osdmap e396: 8 total, 8 up, 8 in 2024-09-10T19:10:19.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:19.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:19.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:19 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:19.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:19.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:19 smithi119 ceph-mon[40674]: osdmap e397: 8 total, 8 up, 8 in 2024-09-10T19:10:19.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:19 smithi194 ceph-mon[32464]: osdmap e396: 8 total, 8 up, 8 in 2024-09-10T19:10:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:19 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:19 smithi194 ceph-mon[32464]: osdmap e397: 8 total, 8 up, 8 in 2024-09-10T19:10:20.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:20 smithi194 ceph-mon[32464]: pgmap v579: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:20.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:20 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:20.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:20.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:10:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:10:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:10:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:20 smithi119 ceph-mon[29577]: pgmap v579: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:20 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:20 smithi119 ceph-mon[40674]: pgmap v579: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:20 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:21.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:21 smithi194 ceph-mon[32464]: osdmap e398: 8 total, 8 up, 8 in 2024-09-10T19:10:21.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:21 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-67"}]: dispatch 2024-09-10T19:10:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:21 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-67"}]: dispatch 2024-09-10T19:10:21.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:21 smithi119 ceph-mon[29577]: osdmap e398: 8 total, 8 up, 8 in 2024-09-10T19:10:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:21 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-67"}]: dispatch 2024-09-10T19:10:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:21 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-67"}]: dispatch 2024-09-10T19:10:21.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:21 smithi119 ceph-mon[40674]: osdmap e398: 8 total, 8 up, 8 in 2024-09-10T19:10:21.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:21.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:21 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:21.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-67"}]: dispatch 2024-09-10T19:10:21.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:21 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-67"}]: dispatch 2024-09-10T19:10:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:22 smithi119 ceph-mon[29577]: pgmap v582: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:22 smithi119 ceph-mon[29577]: osdmap e399: 8 total, 8 up, 8 in 2024-09-10T19:10:22.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:22 smithi119 ceph-mon[40674]: pgmap v582: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:22.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:22 smithi119 ceph-mon[40674]: osdmap e399: 8 total, 8 up, 8 in 2024-09-10T19:10:22.772 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:22 smithi194 ceph-mon[32464]: pgmap v582: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:22.772 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:22 smithi194 ceph-mon[32464]: osdmap e399: 8 total, 8 up, 8 in 2024-09-10T19:10:23.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:10:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:10:22] "GET /metrics HTTP/1.1" 200 228483 "" "Prometheus/2.43.0" 2024-09-10T19:10:23.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:23 smithi194 ceph-mon[32464]: osdmap e400: 8 total, 8 up, 8 in 2024-09-10T19:10:23.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:23.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:23 smithi119 ceph-mon[29577]: osdmap e400: 8 total, 8 up, 8 in 2024-09-10T19:10:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:23.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:23 smithi119 ceph-mon[40674]: osdmap e400: 8 total, 8 up, 8 in 2024-09-10T19:10:23.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:23.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:24.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:24 smithi194 ceph-mon[32464]: pgmap v585: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:10:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:24 smithi194 ceph-mon[32464]: osdmap e401: 8 total, 8 up, 8 in 2024-09-10T19:10:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-69"}]: dispatch 2024-09-10T19:10:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-69"}]: dispatch 2024-09-10T19:10:24.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[40674]: pgmap v585: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:10:24.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:24.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[40674]: osdmap e401: 8 total, 8 up, 8 in 2024-09-10T19:10:24.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:24.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:24.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:24.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-69"}]: dispatch 2024-09-10T19:10:24.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-69"}]: dispatch 2024-09-10T19:10:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[29577]: pgmap v585: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:10:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[29577]: osdmap e401: 8 total, 8 up, 8 in 2024-09-10T19:10:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-69"}]: dispatch 2024-09-10T19:10:24.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-69"}]: dispatch 2024-09-10T19:10:25.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:25 smithi194 ceph-mon[32464]: osdmap e402: 8 total, 8 up, 8 in 2024-09-10T19:10:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:25 smithi119 ceph-mon[29577]: osdmap e402: 8 total, 8 up, 8 in 2024-09-10T19:10:25.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:25 smithi119 ceph-mon[40674]: osdmap e402: 8 total, 8 up, 8 in 2024-09-10T19:10:26.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:26 smithi194 ceph-mon[32464]: pgmap v588: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:10:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:26 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:26 smithi194 ceph-mon[32464]: osdmap e403: 8 total, 8 up, 8 in 2024-09-10T19:10:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:26 smithi119 ceph-mon[29577]: pgmap v588: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:10:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:26 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:26 smithi119 ceph-mon[29577]: osdmap e403: 8 total, 8 up, 8 in 2024-09-10T19:10:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:26 smithi119 ceph-mon[40674]: pgmap v588: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:10:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:26 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:26 smithi119 ceph-mon[40674]: osdmap e403: 8 total, 8 up, 8 in 2024-09-10T19:10:26.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:26.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:27.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:27 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:27 smithi194 ceph-mon[32464]: osdmap e404: 8 total, 8 up, 8 in 2024-09-10T19:10:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:10:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:27 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:10:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:27 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:27 smithi119 ceph-mon[29577]: osdmap e404: 8 total, 8 up, 8 in 2024-09-10T19:10:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:10:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:27 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:10:27.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:27 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:27.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:27 smithi119 ceph-mon[40674]: osdmap e404: 8 total, 8 up, 8 in 2024-09-10T19:10:27.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:27.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:10:27.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:27 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:10:28.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:28 smithi194 ceph-mon[32464]: pgmap v591: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:28.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:10:28.670 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:28 smithi194 ceph-mon[32464]: osdmap e405: 8 total, 8 up, 8 in 2024-09-10T19:10:28.670 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:10:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:10:28.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:28 smithi119 ceph-mon[29577]: pgmap v591: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:10:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:28 smithi119 ceph-mon[29577]: osdmap e405: 8 total, 8 up, 8 in 2024-09-10T19:10:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:28 smithi119 ceph-mon[40674]: pgmap v591: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:10:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:28 smithi119 ceph-mon[40674]: osdmap e405: 8 total, 8 up, 8 in 2024-09-10T19:10:29.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:29 smithi194 ceph-mon[32464]: osdmap e406: 8 total, 8 up, 8 in 2024-09-10T19:10:29.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:29 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:29 smithi119 ceph-mon[29577]: osdmap e406: 8 total, 8 up, 8 in 2024-09-10T19:10:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:29 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:29.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:29 smithi119 ceph-mon[40674]: osdmap e406: 8 total, 8 up, 8 in 2024-09-10T19:10:29.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:29 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:30.414 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:10:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:10:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:10:30.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:30 smithi194 ceph-mon[32464]: pgmap v594: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-09-10T19:10:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:30 smithi119 ceph-mon[29577]: pgmap v594: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-09-10T19:10:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:30 smithi119 ceph-mon[40674]: pgmap v594: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-09-10T19:10:32.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:32 smithi119 ceph-mon[29577]: pgmap v595: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.1 KiB/s wr, 4 op/s 2024-09-10T19:10:32.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:32.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:32 smithi119 ceph-mon[40674]: pgmap v595: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.1 KiB/s wr, 4 op/s 2024-09-10T19:10:32.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:32 smithi194 ceph-mon[32464]: pgmap v595: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.1 KiB/s wr, 4 op/s 2024-09-10T19:10:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:33.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:10:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:10:32] "GET /metrics HTTP/1.1" 200 228501 "" "Prometheus/2.43.0" 2024-09-10T19:10:34.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:34 smithi194 ceph-mon[32464]: pgmap v596: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-09-10T19:10:34.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:34 smithi119 ceph-mon[29577]: pgmap v596: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-09-10T19:10:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:34 smithi119 ceph-mon[40674]: pgmap v596: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2024-09-10T19:10:36.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:36 smithi194 ceph-mon[32464]: pgmap v597: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-09-10T19:10:36.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:36 smithi119 ceph-mon[29577]: pgmap v597: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-09-10T19:10:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:36 smithi119 ceph-mon[40674]: pgmap v597: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.0 KiB/s wr, 4 op/s 2024-09-10T19:10:38.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:38 smithi194 ceph-mon[32464]: pgmap v598: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-10T19:10:38.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:10:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:10:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:38 smithi119 ceph-mon[29577]: pgmap v598: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-10T19:10:38.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:38 smithi119 ceph-mon[40674]: pgmap v598: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-10T19:10:39.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:39 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:39.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:39 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:39.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:39 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:40.439 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:10:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:10:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:10:40.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:40 smithi194 ceph-mon[32464]: pgmap v599: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 379 B/s wr, 1 op/s 2024-09-10T19:10:40.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:40 smithi119 ceph-mon[29577]: pgmap v599: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 379 B/s wr, 1 op/s 2024-09-10T19:10:40.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:40 smithi119 ceph-mon[40674]: pgmap v599: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 379 B/s wr, 1 op/s 2024-09-10T19:10:42.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:42 smithi119 ceph-mon[29577]: pgmap v600: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 341 B/s wr, 1 op/s 2024-09-10T19:10:42.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:42 smithi119 ceph-mon[40674]: pgmap v600: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 341 B/s wr, 1 op/s 2024-09-10T19:10:42.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:42 smithi194 ceph-mon[32464]: pgmap v600: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 341 B/s wr, 1 op/s 2024-09-10T19:10:43.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:10:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:10:42] "GET /metrics HTTP/1.1" 200 228501 "" "Prometheus/2.43.0" 2024-09-10T19:10:44.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:44 smithi194 ceph-mon[32464]: pgmap v601: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 597 B/s wr, 3 op/s 2024-09-10T19:10:44.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:44 smithi119 ceph-mon[29577]: pgmap v601: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 597 B/s wr, 3 op/s 2024-09-10T19:10:44.909 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:44 smithi119 ceph-mon[40674]: pgmap v601: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 597 B/s wr, 3 op/s 2024-09-10T19:10:45.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:45 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:45 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:45 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:45 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:10:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:45 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:45 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:45 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:45 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:10:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:45 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:45 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:45 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:45 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:10:46.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:46 smithi194 ceph-mon[32464]: pgmap v602: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:10:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:46 smithi119 ceph-mon[29577]: pgmap v602: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:10:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:46 smithi119 ceph-mon[40674]: pgmap v602: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:10:47.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:47.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:47.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:48.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:10:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:10:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:48 smithi194 ceph-mon[32464]: pgmap v603: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:10:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-71"}]: dispatch 2024-09-10T19:10:48.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-71"}]: dispatch 2024-09-10T19:10:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:48 smithi119 ceph-mon[29577]: pgmap v603: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:10:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-71"}]: dispatch 2024-09-10T19:10:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-71"}]: dispatch 2024-09-10T19:10:48.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:48 smithi119 ceph-mon[40674]: pgmap v603: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:10:48.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:48.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:10:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-71"}]: dispatch 2024-09-10T19:10:48.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-71"}]: dispatch 2024-09-10T19:10:49.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:49 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:49.718 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:49 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:49.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:49 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:50.465 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:10:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:10:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:10:50.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:50 smithi194 ceph-mon[32464]: pgmap v604: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:10:50.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:50 smithi194 ceph-mon[32464]: osdmap e407: 8 total, 8 up, 8 in 2024-09-10T19:10:50.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:50 smithi119 ceph-mon[29577]: pgmap v604: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:10:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:50 smithi119 ceph-mon[29577]: osdmap e407: 8 total, 8 up, 8 in 2024-09-10T19:10:50.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:50 smithi119 ceph-mon[40674]: pgmap v604: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:10:50.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:50 smithi119 ceph-mon[40674]: osdmap e407: 8 total, 8 up, 8 in 2024-09-10T19:10:51.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:51 smithi194 ceph-mon[32464]: osdmap e408: 8 total, 8 up, 8 in 2024-09-10T19:10:51.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:51.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:51 smithi119 ceph-mon[29577]: osdmap e408: 8 total, 8 up, 8 in 2024-09-10T19:10:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:51 smithi119 ceph-mon[40674]: osdmap e408: 8 total, 8 up, 8 in 2024-09-10T19:10:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:10:52.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:52 smithi194 ceph-mon[32464]: pgmap v607: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:52 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:52 smithi194 ceph-mon[32464]: osdmap e409: 8 total, 8 up, 8 in 2024-09-10T19:10:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:52.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:10:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:10:52] "GET /metrics HTTP/1.1" 200 228467 "" "Prometheus/2.43.0" 2024-09-10T19:10:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:52 smithi119 ceph-mon[29577]: pgmap v607: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:52 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:52 smithi119 ceph-mon[29577]: osdmap e409: 8 total, 8 up, 8 in 2024-09-10T19:10:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:52.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:52 smithi119 ceph-mon[40674]: pgmap v607: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:52.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:52 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:52.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:10:52.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:52 smithi119 ceph-mon[40674]: osdmap e409: 8 total, 8 up, 8 in 2024-09-10T19:10:52.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:10:53.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:53 smithi194 ceph-mon[32464]: osdmap e410: 8 total, 8 up, 8 in 2024-09-10T19:10:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:53 smithi119 ceph-mon[40674]: osdmap e410: 8 total, 8 up, 8 in 2024-09-10T19:10:53.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:53 smithi119 ceph-mon[29577]: osdmap e410: 8 total, 8 up, 8 in 2024-09-10T19:10:54.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:54 smithi194 ceph-mon[32464]: pgmap v610: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:54 smithi194 ceph-mon[32464]: osdmap e411: 8 total, 8 up, 8 in 2024-09-10T19:10:54.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:54 smithi119 ceph-mon[40674]: pgmap v610: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:54 smithi119 ceph-mon[40674]: osdmap e411: 8 total, 8 up, 8 in 2024-09-10T19:10:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:54 smithi119 ceph-mon[29577]: pgmap v610: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:54 smithi119 ceph-mon[29577]: osdmap e411: 8 total, 8 up, 8 in 2024-09-10T19:10:56.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:56 smithi194 ceph-mon[32464]: pgmap v612: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-10T19:10:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:56 smithi119 ceph-mon[29577]: pgmap v612: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-10T19:10:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:56 smithi119 ceph-mon[40674]: pgmap v612: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 2 op/s 2024-09-10T19:10:57.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:57 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:57 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:57.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:57 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:58.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:58 smithi194 ceph-mon[32464]: pgmap v613: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 1 op/s 2024-09-10T19:10:58.769 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:10:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:10:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:58 smithi119 ceph-mon[29577]: pgmap v613: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 1 op/s 2024-09-10T19:10:58.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:58 smithi119 ceph-mon[40674]: pgmap v613: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 1023 B/s wr, 1 op/s 2024-09-10T19:10:59.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:10:59 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:10:59 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:59.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:10:59 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:00.509 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:11:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:11:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:00.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:00 smithi194 ceph-mon[32464]: pgmap v614: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 929 B/s wr, 4 op/s 2024-09-10T19:11:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:00 smithi119 ceph-mon[29577]: pgmap v614: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 929 B/s wr, 4 op/s 2024-09-10T19:11:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:00 smithi119 ceph-mon[40674]: pgmap v614: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 929 B/s wr, 4 op/s 2024-09-10T19:11:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:02 smithi194 ceph-mon[32464]: pgmap v615: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 823 B/s wr, 3 op/s 2024-09-10T19:11:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:02 smithi119 ceph-mon[29577]: pgmap v615: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 823 B/s wr, 3 op/s 2024-09-10T19:11:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:02.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:11:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:11:02] "GET /metrics HTTP/1.1" 200 228488 "" "Prometheus/2.43.0" 2024-09-10T19:11:02.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:02 smithi119 ceph-mon[40674]: pgmap v615: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 823 B/s wr, 3 op/s 2024-09-10T19:11:02.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:05.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:04 smithi194 ceph-mon[32464]: pgmap v616: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 102 B/s wr, 2 op/s 2024-09-10T19:11:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-73"}]: dispatch 2024-09-10T19:11:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-73"}]: dispatch 2024-09-10T19:11:05.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:04 smithi119 ceph-mon[29577]: pgmap v616: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 102 B/s wr, 2 op/s 2024-09-10T19:11:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-73"}]: dispatch 2024-09-10T19:11:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-73"}]: dispatch 2024-09-10T19:11:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:04 smithi119 ceph-mon[40674]: pgmap v616: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 102 B/s wr, 2 op/s 2024-09-10T19:11:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-73"}]: dispatch 2024-09-10T19:11:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-73"}]: dispatch 2024-09-10T19:11:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:05 smithi119 ceph-mon[29577]: osdmap e412: 8 total, 8 up, 8 in 2024-09-10T19:11:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:05 smithi119 ceph-mon[29577]: pgmap v618: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:05 smithi119 ceph-mon[29577]: osdmap e413: 8 total, 8 up, 8 in 2024-09-10T19:11:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:06.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:05 smithi119 ceph-mon[40674]: osdmap e412: 8 total, 8 up, 8 in 2024-09-10T19:11:06.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:05 smithi119 ceph-mon[40674]: pgmap v618: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:06.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:05 smithi119 ceph-mon[40674]: osdmap e413: 8 total, 8 up, 8 in 2024-09-10T19:11:06.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:06.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:06.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:05 smithi194 ceph-mon[32464]: osdmap e412: 8 total, 8 up, 8 in 2024-09-10T19:11:06.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:05 smithi194 ceph-mon[32464]: pgmap v618: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:06.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:05 smithi194 ceph-mon[32464]: osdmap e413: 8 total, 8 up, 8 in 2024-09-10T19:11:06.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:06.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:07 smithi119 ceph-mon[29577]: osdmap e414: 8 total, 8 up, 8 in 2024-09-10T19:11:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:07.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:07 smithi119 ceph-mon[40674]: osdmap e414: 8 total, 8 up, 8 in 2024-09-10T19:11:07.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:07 smithi194 ceph-mon[32464]: osdmap e414: 8 total, 8 up, 8 in 2024-09-10T19:11:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:08 smithi119 ceph-mon[29577]: pgmap v621: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:08 smithi119 ceph-mon[29577]: osdmap e415: 8 total, 8 up, 8 in 2024-09-10T19:11:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:08 smithi119 ceph-mon[40674]: pgmap v621: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:08 smithi119 ceph-mon[40674]: osdmap e415: 8 total, 8 up, 8 in 2024-09-10T19:11:08.818 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:11:08 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2[56942]: 2024-09-10T19:11:08.561+0000 7f693ae63640 -1 snap_mapper.add_oid found existing snaps mapped on 63:563dedaa:test-rados-api-smithi119-79782-76::foo:28, removing 2024-09-10T19:11:09.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:11:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:11:09.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:08 smithi194 ceph-mon[32464]: pgmap v621: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:09.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:08 smithi194 ceph-mon[32464]: osdmap e415: 8 total, 8 up, 8 in 2024-09-10T19:11:09.019 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:11:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:11:08.562+0000 7f087fc9a640 -1 snap_mapper.add_oid found existing snaps mapped on 63:563dedaa:test-rados-api-smithi119-79782-76::foo:28, removing 2024-09-10T19:11:09.020 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:11:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:11:08.561+0000 7f2e11dcb640 -1 snap_mapper.add_oid found existing snaps mapped on 63:563dedaa:test-rados-api-smithi119-79782-76::foo:28, removing 2024-09-10T19:11:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:09 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:09 smithi119 ceph-mon[29577]: osdmap e416: 8 total, 8 up, 8 in 2024-09-10T19:11:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-75"}]: dispatch 2024-09-10T19:11:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-75"}]: dispatch 2024-09-10T19:11:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:09 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:09 smithi119 ceph-mon[40674]: osdmap e416: 8 total, 8 up, 8 in 2024-09-10T19:11:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-75"}]: dispatch 2024-09-10T19:11:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-75"}]: dispatch 2024-09-10T19:11:10.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:09 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:10.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:09 smithi194 ceph-mon[32464]: osdmap e416: 8 total, 8 up, 8 in 2024-09-10T19:11:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-75"}]: dispatch 2024-09-10T19:11:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-75"}]: dispatch 2024-09-10T19:11:10.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:11:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:11:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:10.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:10 smithi119 ceph-mon[29577]: pgmap v624: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:11:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:10 smithi119 ceph-mon[29577]: osdmap e417: 8 total, 8 up, 8 in 2024-09-10T19:11:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:10 smithi119 ceph-mon[40674]: pgmap v624: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:11:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:10 smithi119 ceph-mon[40674]: osdmap e417: 8 total, 8 up, 8 in 2024-09-10T19:11:11.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:10 smithi194 ceph-mon[32464]: pgmap v624: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:11:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:10 smithi194 ceph-mon[32464]: osdmap e417: 8 total, 8 up, 8 in 2024-09-10T19:11:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:11 smithi119 ceph-mon[40674]: osdmap e418: 8 total, 8 up, 8 in 2024-09-10T19:11:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:11 smithi119 ceph-mon[40674]: osdmap e419: 8 total, 8 up, 8 in 2024-09-10T19:11:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:11 smithi119 ceph-mon[29577]: osdmap e418: 8 total, 8 up, 8 in 2024-09-10T19:11:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:11 smithi119 ceph-mon[29577]: osdmap e419: 8 total, 8 up, 8 in 2024-09-10T19:11:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:12.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:11 smithi194 ceph-mon[32464]: osdmap e418: 8 total, 8 up, 8 in 2024-09-10T19:11:12.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:12.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:12.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:12.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:11 smithi194 ceph-mon[32464]: osdmap e419: 8 total, 8 up, 8 in 2024-09-10T19:11:12.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:12 smithi119 ceph-mon[29577]: pgmap v627: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:11:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:12 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:11:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:12 smithi119 ceph-mon[29577]: osdmap e420: 8 total, 8 up, 8 in 2024-09-10T19:11:12.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:11:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:11:12] "GET /metrics HTTP/1.1" 200 228488 "" "Prometheus/2.43.0" 2024-09-10T19:11:12.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:12 smithi119 ceph-mon[40674]: pgmap v627: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:11:12.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:12 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:11:12.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:12 smithi119 ceph-mon[40674]: osdmap e420: 8 total, 8 up, 8 in 2024-09-10T19:11:12.818 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:11:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0[44998]: 2024-09-10T19:11:12.567+0000 7f953f4dc640 -1 snap_mapper.add_oid found existing snaps mapped on 63:05124412:test-rados-api-smithi119-79782-78::foo:29, removing 2024-09-10T19:11:12.818 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:11:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1[50994]: 2024-09-10T19:11:12.567+0000 7f7565b75640 -1 snap_mapper.add_oid found existing snaps mapped on 63:05124412:test-rados-api-smithi119-79782-78::foo:29, removing 2024-09-10T19:11:13.018 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:11:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:11:12.567+0000 7f3c1d2ea640 -1 snap_mapper.add_oid found existing snaps mapped on 63:05124412:test-rados-api-smithi119-79782-78::foo:29, removing 2024-09-10T19:11:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:12 smithi194 ceph-mon[32464]: pgmap v627: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:11:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:12 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:11:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:12 smithi194 ceph-mon[32464]: osdmap e420: 8 total, 8 up, 8 in 2024-09-10T19:11:14.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:14.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:14.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-77"}]: dispatch 2024-09-10T19:11:14.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-77"}]: dispatch 2024-09-10T19:11:14.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:14.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:14.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-77"}]: dispatch 2024-09-10T19:11:14.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-77"}]: dispatch 2024-09-10T19:11:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-77"}]: dispatch 2024-09-10T19:11:14.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-77"}]: dispatch 2024-09-10T19:11:15.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:14 smithi194 ceph-mon[32464]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-09-10T19:11:15.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:14 smithi194 ceph-mon[32464]: osdmap e421: 8 total, 8 up, 8 in 2024-09-10T19:11:15.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:14 smithi119 ceph-mon[29577]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-09-10T19:11:15.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:14 smithi119 ceph-mon[29577]: osdmap e421: 8 total, 8 up, 8 in 2024-09-10T19:11:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:14 smithi119 ceph-mon[40674]: pgmap v630: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.5 KiB/s wr, 8 op/s 2024-09-10T19:11:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:14 smithi119 ceph-mon[40674]: osdmap e421: 8 total, 8 up, 8 in 2024-09-10T19:11:16.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:15 smithi194 ceph-mon[32464]: osdmap e422: 8 total, 8 up, 8 in 2024-09-10T19:11:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:16.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:15 smithi119 ceph-mon[29577]: osdmap e422: 8 total, 8 up, 8 in 2024-09-10T19:11:16.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:16.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:16.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:15 smithi119 ceph-mon[40674]: osdmap e422: 8 total, 8 up, 8 in 2024-09-10T19:11:16.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:16.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:17.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:16 smithi194 ceph-mon[32464]: pgmap v633: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-09-10T19:11:17.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:17.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:16 smithi194 ceph-mon[32464]: osdmap e423: 8 total, 8 up, 8 in 2024-09-10T19:11:17.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:17.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:16 smithi119 ceph-mon[29577]: pgmap v633: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-09-10T19:11:17.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:17.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:16 smithi119 ceph-mon[29577]: osdmap e423: 8 total, 8 up, 8 in 2024-09-10T19:11:17.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:17.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:16 smithi119 ceph-mon[40674]: pgmap v633: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.7 KiB/s wr, 5 op/s 2024-09-10T19:11:17.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:17.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:16 smithi119 ceph-mon[40674]: osdmap e423: 8 total, 8 up, 8 in 2024-09-10T19:11:17.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:18.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:17 smithi194 ceph-mon[32464]: osdmap e424: 8 total, 8 up, 8 in 2024-09-10T19:11:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:18.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:17 smithi119 ceph-mon[29577]: osdmap e424: 8 total, 8 up, 8 in 2024-09-10T19:11:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:18.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:17 smithi119 ceph-mon[40674]: osdmap e424: 8 total, 8 up, 8 in 2024-09-10T19:11:18.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:19.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:11:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:11:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:18 smithi194 ceph-mon[32464]: pgmap v636: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:18 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:11:19.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:18 smithi194 ceph-mon[32464]: osdmap e425: 8 total, 8 up, 8 in 2024-09-10T19:11:19.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.d"}]: dispatch 2024-09-10T19:11:19.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:18 smithi194 ceph-mon[32464]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.d"}]: dispatch 2024-09-10T19:11:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:18 smithi119 ceph-mon[29577]: pgmap v636: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:18 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:11:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:18 smithi119 ceph-mon[29577]: osdmap e425: 8 total, 8 up, 8 in 2024-09-10T19:11:19.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.d"}]: dispatch 2024-09-10T19:11:19.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:18 smithi119 ceph-mon[29577]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.d"}]: dispatch 2024-09-10T19:11:19.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:18 smithi119 ceph-mon[40674]: pgmap v636: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:19.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:18 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:11:19.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:18 smithi119 ceph-mon[40674]: osdmap e425: 8 total, 8 up, 8 in 2024-09-10T19:11:19.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.d"}]: dispatch 2024-09-10T19:11:19.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:18 smithi119 ceph-mon[40674]: from='mon.? -' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "223.d"}]: dispatch 2024-09-10T19:11:19.968 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:19 smithi119 ceph-mon[29577]: 223.d deep-scrub starts 2024-09-10T19:11:19.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:19 smithi119 ceph-mon[29577]: 223.d deep-scrub ok 2024-09-10T19:11:19.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:19 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:19.969 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:19 smithi119 ceph-mon[40674]: 223.d deep-scrub starts 2024-09-10T19:11:19.969 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:19 smithi119 ceph-mon[40674]: 223.d deep-scrub ok 2024-09-10T19:11:19.969 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:19 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:20.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:19 smithi194 ceph-mon[32464]: 223.d deep-scrub starts 2024-09-10T19:11:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:19 smithi194 ceph-mon[32464]: 223.d deep-scrub ok 2024-09-10T19:11:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:19 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:20.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:11:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:11:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:21.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:20 smithi194 ceph-mon[32464]: pgmap v638: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-10T19:11:21.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:20 smithi119 ceph-mon[29577]: pgmap v638: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-10T19:11:21.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:20 smithi119 ceph-mon[40674]: pgmap v638: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 3 op/s 2024-09-10T19:11:23.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:22 smithi194 ceph-mon[32464]: pgmap v639: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-09-10T19:11:23.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:22 smithi119 ceph-mon[29577]: pgmap v639: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-09-10T19:11:23.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:11:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:11:22] "GET /metrics HTTP/1.1" 200 228487 "" "Prometheus/2.43.0" 2024-09-10T19:11:23.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:22 smithi119 ceph-mon[40674]: pgmap v639: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-09-10T19:11:25.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:24 smithi194 ceph-mon[32464]: pgmap v640: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 672 B/s wr, 2 op/s 2024-09-10T19:11:25.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:24 smithi119 ceph-mon[29577]: pgmap v640: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 672 B/s wr, 2 op/s 2024-09-10T19:11:25.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:24 smithi119 ceph-mon[40674]: pgmap v640: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 672 B/s wr, 2 op/s 2024-09-10T19:11:27.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:26 smithi194 ceph-mon[32464]: pgmap v641: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 594 B/s wr, 2 op/s 2024-09-10T19:11:27.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:26 smithi119 ceph-mon[29577]: pgmap v641: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 594 B/s wr, 2 op/s 2024-09-10T19:11:27.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:26 smithi119 ceph-mon[40674]: pgmap v641: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 594 B/s wr, 2 op/s 2024-09-10T19:11:28.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:11:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:11:28.821 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:28 smithi194 ceph-mon[32464]: pgmap v642: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:11:29.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:28 smithi119 ceph-mon[29577]: pgmap v642: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:11:29.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:28 smithi119 ceph-mon[40674]: pgmap v642: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:11:30.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:29 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:30.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:29 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:30.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:29 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:30.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:11:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:11:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:31.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:30 smithi194 ceph-mon[32464]: pgmap v643: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 441 B/s wr, 2 op/s 2024-09-10T19:11:31.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:30 smithi119 ceph-mon[29577]: pgmap v643: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 441 B/s wr, 2 op/s 2024-09-10T19:11:31.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:30 smithi119 ceph-mon[40674]: pgmap v643: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 441 B/s wr, 2 op/s 2024-09-10T19:11:33.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:32 smithi194 ceph-mon[32464]: pgmap v644: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:33.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:32 smithi119 ceph-mon[29577]: pgmap v644: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:33.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:33.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:11:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:11:32] "GET /metrics HTTP/1.1" 200 228462 "" "Prometheus/2.43.0" 2024-09-10T19:11:33.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:32 smithi119 ceph-mon[40674]: pgmap v644: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:33.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:34 smithi194 ceph-mon[32464]: pgmap v645: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:35.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:34 smithi119 ceph-mon[29577]: pgmap v645: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:35.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:34 smithi119 ceph-mon[40674]: pgmap v645: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:37.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:36 smithi194 ceph-mon[32464]: pgmap v646: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:37.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:36 smithi119 ceph-mon[29577]: pgmap v646: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:37.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:36 smithi119 ceph-mon[40674]: pgmap v646: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:38.018 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:11:37 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:11:37.629+0000 7f08834a1640 -1 snap_mapper.add_oid found existing snaps mapped on 223:b33f392f:test-rados-api-smithi119-79782-80::foo:2, removing 2024-09-10T19:11:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:37 smithi194 ceph-mon[32464]: pgmap v647: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-79"}]: dispatch 2024-09-10T19:11:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-79"}]: dispatch 2024-09-10T19:11:38.066 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:11:37 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0[44998]: 2024-09-10T19:11:37.629+0000 7f9540cdf640 -1 snap_mapper.add_oid found existing snaps mapped on 223:b33f392f:test-rados-api-smithi119-79782-80::foo:2, removing 2024-09-10T19:11:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:37 smithi119 ceph-mon[40674]: pgmap v647: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-79"}]: dispatch 2024-09-10T19:11:38.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-79"}]: dispatch 2024-09-10T19:11:38.067 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:11:37 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1[50994]: 2024-09-10T19:11:37.629+0000 7f7560b6b640 -1 snap_mapper.add_oid found existing snaps mapped on 223:b33f392f:test-rados-api-smithi119-79782-80::foo:2, removing 2024-09-10T19:11:38.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:37 smithi119 ceph-mon[29577]: pgmap v647: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:38.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:38.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:38.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-79"}]: dispatch 2024-09-10T19:11:38.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-79"}]: dispatch 2024-09-10T19:11:39.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:11:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:11:40.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:39 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:40.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:39 smithi194 ceph-mon[32464]: osdmap e426: 8 total, 8 up, 8 in 2024-09-10T19:11:40.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:39 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:40.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:39 smithi119 ceph-mon[29577]: osdmap e426: 8 total, 8 up, 8 in 2024-09-10T19:11:40.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:39 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:40.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:39 smithi119 ceph-mon[40674]: osdmap e426: 8 total, 8 up, 8 in 2024-09-10T19:11:40.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:11:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:11:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:40 smithi194 ceph-mon[32464]: pgmap v649: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:11:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:40 smithi194 ceph-mon[32464]: osdmap e427: 8 total, 8 up, 8 in 2024-09-10T19:11:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:40 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:40 smithi119 ceph-mon[29577]: pgmap v649: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:11:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:40 smithi119 ceph-mon[29577]: osdmap e427: 8 total, 8 up, 8 in 2024-09-10T19:11:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:40 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:40 smithi119 ceph-mon[40674]: pgmap v649: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:11:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:40 smithi119 ceph-mon[40674]: osdmap e427: 8 total, 8 up, 8 in 2024-09-10T19:11:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:40 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:42.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:41 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:41 smithi194 ceph-mon[32464]: osdmap e428: 8 total, 8 up, 8 in 2024-09-10T19:11:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:11:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:41 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:11:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:41 smithi194 ceph-mon[32464]: pgmap v652: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:11:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:41 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:41 smithi119 ceph-mon[29577]: osdmap e428: 8 total, 8 up, 8 in 2024-09-10T19:11:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:11:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:41 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:11:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:41 smithi119 ceph-mon[29577]: pgmap v652: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:11:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:41 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:41 smithi119 ceph-mon[40674]: osdmap e428: 8 total, 8 up, 8 in 2024-09-10T19:11:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:11:42.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:41 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:11:42.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:41 smithi119 ceph-mon[40674]: pgmap v652: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:11:43.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:11:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:42 smithi194 ceph-mon[32464]: osdmap e429: 8 total, 8 up, 8 in 2024-09-10T19:11:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:42 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:43.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:11:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:11:42] "GET /metrics HTTP/1.1" 200 228462 "" "Prometheus/2.43.0" 2024-09-10T19:11:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:11:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:42 smithi119 ceph-mon[29577]: osdmap e429: 8 total, 8 up, 8 in 2024-09-10T19:11:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:42 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:43.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:11:43.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:42 smithi119 ceph-mon[40674]: osdmap e429: 8 total, 8 up, 8 in 2024-09-10T19:11:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:42 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:44.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:11:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:43 smithi194 ceph-mon[32464]: osdmap e430: 8 total, 8 up, 8 in 2024-09-10T19:11:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:11:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:43 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:11:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:43 smithi194 ceph-mon[32464]: pgmap v655: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:44.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:11:44.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:43 smithi119 ceph-mon[29577]: osdmap e430: 8 total, 8 up, 8 in 2024-09-10T19:11:44.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:11:44.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:43 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:11:44.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:43 smithi119 ceph-mon[29577]: pgmap v655: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:44.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:11:44.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:43 smithi119 ceph-mon[40674]: osdmap e430: 8 total, 8 up, 8 in 2024-09-10T19:11:44.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:11:44.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:43 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:11:44.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:43 smithi119 ceph-mon[40674]: pgmap v655: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:45.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:11:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:44 smithi194 ceph-mon[32464]: osdmap e431: 8 total, 8 up, 8 in 2024-09-10T19:11:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:44 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:11:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:44 smithi194 ceph-mon[32464]: osdmap e432: 8 total, 8 up, 8 in 2024-09-10T19:11:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:11:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:44 smithi119 ceph-mon[40674]: osdmap e431: 8 total, 8 up, 8 in 2024-09-10T19:11:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:44 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:11:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:44 smithi119 ceph-mon[40674]: osdmap e432: 8 total, 8 up, 8 in 2024-09-10T19:11:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:11:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:44 smithi119 ceph-mon[29577]: osdmap e431: 8 total, 8 up, 8 in 2024-09-10T19:11:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:45.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:44 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:11:45.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:44 smithi119 ceph-mon[29577]: osdmap e432: 8 total, 8 up, 8 in 2024-09-10T19:11:46.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-09-10T19:11:46.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:45 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-09-10T19:11:46.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:45 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:11:46.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:45 smithi194 ceph-mon[32464]: pgmap v658: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:46.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:45 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:11:46.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:45 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:11:46.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:45 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:11:46.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-09-10T19:11:46.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:45 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-09-10T19:11:46.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:45 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:11:46.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:45 smithi119 ceph-mon[29577]: pgmap v658: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:46.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:45 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:11:46.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:45 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:11:46.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:45 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:11:46.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-09-10T19:11:46.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:45 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-09-10T19:11:46.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:45 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:11:46.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:45 smithi119 ceph-mon[40674]: pgmap v658: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:46.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:45 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:11:46.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:45 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:11:46.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:45 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:11:47.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-09-10T19:11:47.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:46 smithi194 ceph-mon[32464]: osdmap e433: 8 total, 8 up, 8 in 2024-09-10T19:11:47.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-09-10T19:11:47.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:46 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-09-10T19:11:47.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-09-10T19:11:47.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:46 smithi119 ceph-mon[29577]: osdmap e433: 8 total, 8 up, 8 in 2024-09-10T19:11:47.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-09-10T19:11:47.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:46 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-09-10T19:11:47.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-09-10T19:11:47.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:46 smithi119 ceph-mon[40674]: osdmap e433: 8 total, 8 up, 8 in 2024-09-10T19:11:47.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-09-10T19:11:47.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:46 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-09-10T19:11:48.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:47 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-09-10T19:11:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:47 smithi194 ceph-mon[32464]: osdmap e434: 8 total, 8 up, 8 in 2024-09-10T19:11:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:47 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:47 smithi194 ceph-mon[32464]: pgmap v661: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:48.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:47 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-09-10T19:11:48.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:47 smithi119 ceph-mon[29577]: osdmap e434: 8 total, 8 up, 8 in 2024-09-10T19:11:48.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:48.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:47 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:48.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:48.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:47 smithi119 ceph-mon[29577]: pgmap v661: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:48.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:47 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-09-10T19:11:48.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:47 smithi119 ceph-mon[40674]: osdmap e434: 8 total, 8 up, 8 in 2024-09-10T19:11:48.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:48.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:47 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:48.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:48.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:47 smithi119 ceph-mon[40674]: pgmap v661: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:48.949 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:11:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:11:49.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:11:49.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:48 smithi194 ceph-mon[32464]: osdmap e435: 8 total, 8 up, 8 in 2024-09-10T19:11:49.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:49.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:49.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-81"}]: dispatch 2024-09-10T19:11:49.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:48 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-81"}]: dispatch 2024-09-10T19:11:49.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:48 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:49.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:11:49.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:48 smithi119 ceph-mon[29577]: osdmap e435: 8 total, 8 up, 8 in 2024-09-10T19:11:49.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:49.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:49.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-81"}]: dispatch 2024-09-10T19:11:49.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:48 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-81"}]: dispatch 2024-09-10T19:11:49.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:48 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:49.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:11:49.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:48 smithi119 ceph-mon[40674]: osdmap e435: 8 total, 8 up, 8 in 2024-09-10T19:11:49.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:49.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:49.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-81"}]: dispatch 2024-09-10T19:11:49.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:48 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-81"}]: dispatch 2024-09-10T19:11:49.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:48 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:50.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:49 smithi119 ceph-mon[29577]: osdmap e436: 8 total, 8 up, 8 in 2024-09-10T19:11:50.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:49 smithi119 ceph-mon[29577]: pgmap v664: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 12 KiB/s wr, 218 op/s 2024-09-10T19:11:50.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:49 smithi119 ceph-mon[29577]: osdmap e437: 8 total, 8 up, 8 in 2024-09-10T19:11:50.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:50.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:49 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:50.218 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:49 smithi119 ceph-mon[40674]: osdmap e436: 8 total, 8 up, 8 in 2024-09-10T19:11:50.218 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:49 smithi119 ceph-mon[40674]: pgmap v664: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 12 KiB/s wr, 218 op/s 2024-09-10T19:11:50.218 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:49 smithi119 ceph-mon[40674]: osdmap e437: 8 total, 8 up, 8 in 2024-09-10T19:11:50.218 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:50.219 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:49 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:50.268 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:11:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:11:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:50.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:49 smithi194 ceph-mon[32464]: osdmap e436: 8 total, 8 up, 8 in 2024-09-10T19:11:50.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:49 smithi194 ceph-mon[32464]: pgmap v664: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 12 KiB/s wr, 218 op/s 2024-09-10T19:11:50.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:49 smithi194 ceph-mon[32464]: osdmap e437: 8 total, 8 up, 8 in 2024-09-10T19:11:50.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:50.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:49 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:11:52.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:52.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:51 smithi194 ceph-mon[32464]: osdmap e438: 8 total, 8 up, 8 in 2024-09-10T19:11:52.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:52.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:11:52.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:51 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:11:52.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:51 smithi194 ceph-mon[32464]: pgmap v667: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 12 KiB/s wr, 218 op/s 2024-09-10T19:11:52.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:52.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:51 smithi119 ceph-mon[29577]: osdmap e438: 8 total, 8 up, 8 in 2024-09-10T19:11:52.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:52.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:11:52.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:51 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:11:52.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:51 smithi119 ceph-mon[29577]: pgmap v667: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 12 KiB/s wr, 218 op/s 2024-09-10T19:11:52.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:11:52.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:51 smithi119 ceph-mon[40674]: osdmap e438: 8 total, 8 up, 8 in 2024-09-10T19:11:52.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:11:52.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:11:52.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:51 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:11:52.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:51 smithi119 ceph-mon[40674]: pgmap v667: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 117 KiB/s rd, 12 KiB/s wr, 218 op/s 2024-09-10T19:11:53.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:11:53.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:52 smithi194 ceph-mon[32464]: osdmap e439: 8 total, 8 up, 8 in 2024-09-10T19:11:53.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:53.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:53.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:11:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:11:52] "GET /metrics HTTP/1.1" 200 228460 "" "Prometheus/2.43.0" 2024-09-10T19:11:53.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:11:53.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:52 smithi119 ceph-mon[29577]: osdmap e439: 8 total, 8 up, 8 in 2024-09-10T19:11:53.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:53.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:53.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:11:53.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:52 smithi119 ceph-mon[40674]: osdmap e439: 8 total, 8 up, 8 in 2024-09-10T19:11:53.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:53.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:54.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:11:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:53 smithi194 ceph-mon[32464]: osdmap e440: 8 total, 8 up, 8 in 2024-09-10T19:11:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:11:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:11:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:53 smithi194 ceph-mon[32464]: pgmap v670: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:54.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:11:54.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:53 smithi119 ceph-mon[29577]: osdmap e440: 8 total, 8 up, 8 in 2024-09-10T19:11:54.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:11:54.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:11:54.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:53 smithi119 ceph-mon[29577]: pgmap v670: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:11:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:53 smithi119 ceph-mon[40674]: osdmap e440: 8 total, 8 up, 8 in 2024-09-10T19:11:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:11:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:11:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:53 smithi119 ceph-mon[40674]: pgmap v670: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:55.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:54 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:11:55.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:54 smithi194 ceph-mon[32464]: osdmap e441: 8 total, 8 up, 8 in 2024-09-10T19:11:55.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:54 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:55.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:54 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:55.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:54 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:11:55.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:54 smithi119 ceph-mon[29577]: osdmap e441: 8 total, 8 up, 8 in 2024-09-10T19:11:55.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:55.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:54 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:55.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:54 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:11:55.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:54 smithi119 ceph-mon[40674]: osdmap e441: 8 total, 8 up, 8 in 2024-09-10T19:11:55.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:54 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:55.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:54 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:11:56.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:55 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:11:56.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:55 smithi194 ceph-mon[32464]: osdmap e442: 8 total, 8 up, 8 in 2024-09-10T19:11:56.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:55 smithi194 ceph-mon[32464]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:55 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:11:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:55 smithi119 ceph-mon[29577]: osdmap e442: 8 total, 8 up, 8 in 2024-09-10T19:11:56.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:55 smithi119 ceph-mon[29577]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:56.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:55 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:11:56.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:55 smithi119 ceph-mon[40674]: osdmap e442: 8 total, 8 up, 8 in 2024-09-10T19:11:56.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:55 smithi119 ceph-mon[40674]: pgmap v673: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:57.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:56 smithi194 ceph-mon[32464]: osdmap e443: 8 total, 8 up, 8 in 2024-09-10T19:11:57.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:56 smithi119 ceph-mon[29577]: osdmap e443: 8 total, 8 up, 8 in 2024-09-10T19:11:57.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:56 smithi119 ceph-mon[40674]: osdmap e443: 8 total, 8 up, 8 in 2024-09-10T19:11:58.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:57 smithi194 ceph-mon[32464]: osdmap e444: 8 total, 8 up, 8 in 2024-09-10T19:11:58.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:58.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:58.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-83"}]: dispatch 2024-09-10T19:11:58.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:57 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-83"}]: dispatch 2024-09-10T19:11:58.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:57 smithi194 ceph-mon[32464]: pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:58.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:57 smithi119 ceph-mon[29577]: osdmap e444: 8 total, 8 up, 8 in 2024-09-10T19:11:58.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:58.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:58.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-83"}]: dispatch 2024-09-10T19:11:58.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:57 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-83"}]: dispatch 2024-09-10T19:11:58.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:57 smithi119 ceph-mon[29577]: pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:58.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:57 smithi119 ceph-mon[40674]: osdmap e444: 8 total, 8 up, 8 in 2024-09-10T19:11:58.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:58.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:11:58.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-83"}]: dispatch 2024-09-10T19:11:58.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:57 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-83"}]: dispatch 2024-09-10T19:11:58.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:57 smithi119 ceph-mon[40674]: pgmap v676: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:58.920 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:58 smithi194 ceph-mon[32464]: osdmap e445: 8 total, 8 up, 8 in 2024-09-10T19:11:58.920 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:58 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:58.920 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:11:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:11:59.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:58 smithi119 ceph-mon[29577]: osdmap e445: 8 total, 8 up, 8 in 2024-09-10T19:11:59.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:58 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:58 smithi119 ceph-mon[40674]: osdmap e445: 8 total, 8 up, 8 in 2024-09-10T19:11:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:58 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:00.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:59 smithi194 ceph-mon[32464]: osdmap e446: 8 total, 8 up, 8 in 2024-09-10T19:12:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:59 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:11:59 smithi194 ceph-mon[32464]: pgmap v679: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-09-10T19:12:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:59 smithi119 ceph-mon[40674]: osdmap e446: 8 total, 8 up, 8 in 2024-09-10T19:12:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:59 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:11:59 smithi119 ceph-mon[40674]: pgmap v679: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-09-10T19:12:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:59 smithi119 ceph-mon[29577]: osdmap e446: 8 total, 8 up, 8 in 2024-09-10T19:12:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:59 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:11:59 smithi119 ceph-mon[29577]: pgmap v679: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-09-10T19:12:00.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:12:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:12:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:00 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:00 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:00 smithi119 ceph-mon[29577]: osdmap e447: 8 total, 8 up, 8 in 2024-09-10T19:12:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:00 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:01.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:00 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:01.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:00 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:01.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:00 smithi119 ceph-mon[40674]: osdmap e447: 8 total, 8 up, 8 in 2024-09-10T19:12:01.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:01.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:01.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:00 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:01.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:00 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:01.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:00 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:01.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:00 smithi194 ceph-mon[32464]: osdmap e447: 8 total, 8 up, 8 in 2024-09-10T19:12:01.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:01.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:01.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:00 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:01 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:12:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:01 smithi119 ceph-mon[29577]: osdmap e448: 8 total, 8 up, 8 in 2024-09-10T19:12:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:01 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:01 smithi119 ceph-mon[29577]: pgmap v682: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-09-10T19:12:02.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:01 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:12:02.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:01 smithi119 ceph-mon[40674]: osdmap e448: 8 total, 8 up, 8 in 2024-09-10T19:12:02.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:02.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:01 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:02.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:01 smithi119 ceph-mon[40674]: pgmap v682: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-09-10T19:12:02.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:01 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:12:02.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:01 smithi194 ceph-mon[32464]: osdmap e448: 8 total, 8 up, 8 in 2024-09-10T19:12:02.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:02.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:01 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:02.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:01 smithi194 ceph-mon[32464]: pgmap v682: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-09-10T19:12:03.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:12:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:12:02] "GET /metrics HTTP/1.1" 200 228518 "" "Prometheus/2.43.0" 2024-09-10T19:12:03.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:02 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:12:03.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:02 smithi119 ceph-mon[29577]: osdmap e449: 8 total, 8 up, 8 in 2024-09-10T19:12:03.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:12:03.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:02 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:12:03.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:03.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:02 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:12:03.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:02 smithi119 ceph-mon[40674]: osdmap e449: 8 total, 8 up, 8 in 2024-09-10T19:12:03.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:12:03.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:02 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:12:03.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:03.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:02 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_tier","val": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:12:03.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:02 smithi194 ceph-mon[32464]: osdmap e449: 8 total, 8 up, 8 in 2024-09-10T19:12:03.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:12:03.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:02 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:12:03.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:04.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:12:04.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:03 smithi119 ceph-mon[29577]: osdmap e450: 8 total, 8 up, 8 in 2024-09-10T19:12:04.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:12:04.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:12:04.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:03 smithi119 ceph-mon[29577]: pgmap v685: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:04.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:12:04.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:03 smithi119 ceph-mon[40674]: osdmap e450: 8 total, 8 up, 8 in 2024-09-10T19:12:04.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:12:04.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:12:04.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:03 smithi119 ceph-mon[40674]: pgmap v685: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:04.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:12:04.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:03 smithi194 ceph-mon[32464]: osdmap e450: 8 total, 8 up, 8 in 2024-09-10T19:12:04.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:12:04.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:12:04.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:03 smithi194 ceph-mon[32464]: pgmap v685: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:05.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:12:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:04 smithi119 ceph-mon[29577]: osdmap e451: 8 total, 8 up, 8 in 2024-09-10T19:12:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:12:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:04 smithi119 ceph-mon[40674]: osdmap e451: 8 total, 8 up, 8 in 2024-09-10T19:12:05.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:12:05.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:04 smithi194 ceph-mon[32464]: osdmap e451: 8 total, 8 up, 8 in 2024-09-10T19:12:06.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:05 smithi119 ceph-mon[29577]: osdmap e452: 8 total, 8 up, 8 in 2024-09-10T19:12:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:05 smithi119 ceph-mon[29577]: pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:06.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:05 smithi119 ceph-mon[40674]: osdmap e452: 8 total, 8 up, 8 in 2024-09-10T19:12:06.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:05 smithi119 ceph-mon[40674]: pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:06.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:05 smithi194 ceph-mon[32464]: osdmap e452: 8 total, 8 up, 8 in 2024-09-10T19:12:06.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:05 smithi194 ceph-mon[32464]: pgmap v688: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:06.818 INFO:tasks.workunit.client.0.smithi119.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18318 ms) 2024-09-10T19:12:06.818 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-10T19:12:06.818 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (21862 ms) 2024-09-10T19:12:06.818 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-10T19:12:06.819 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3016 ms) 2024-09-10T19:12:06.819 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-10T19:12:06.819 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.819 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3023 ms) 2024-09-10T19:12:06.819 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-10T19:12:06.819 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3015 ms) 2024-09-10T19:12:06.819 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-10T19:12:06.819 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.819 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3021 ms) 2024-09-10T19:12:06.819 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-10T19:12:06.819 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.819 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4007 ms) 2024-09-10T19:12:06.820 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-10T19:12:06.820 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.820 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38522 ms) 2024-09-10T19:12:06.820 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-10T19:12:06.820 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.820 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17096 ms) 2024-09-10T19:12:06.820 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-10T19:12:06.820 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.820 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4034 ms) 2024-09-10T19:12:06.820 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-10T19:12:06.820 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.820 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3011 ms) 2024-09-10T19:12:06.820 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-10T19:12:06.821 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.821 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (25086 ms) 2024-09-10T19:12:06.821 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-10T19:12:06.821 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.821 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15051 ms) 2024-09-10T19:12:06.821 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-10T19:12:06.821 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.821 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5027 ms) 2024-09-10T19:12:06.821 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-10T19:12:06.821 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.821 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4032 ms) 2024-09-10T19:12:06.821 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-10T19:12:06.821 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.822 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: waiting for scrubs... 2024-09-10T19:12:06.822 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: done waiting 2024-09-10T19:12:06.822 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25061 ms) 2024-09-10T19:12:06.822 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-10T19:12:06.822 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.822 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10066 ms) 2024-09-10T19:12:06.822 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-10T19:12:06.822 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.822 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9057 ms) 2024-09-10T19:12:06.822 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-10T19:12:06.822 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:12:06.822 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9053 ms) 2024-09-10T19:12:07.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:06 smithi194 ceph-mon[32464]: osdmap e453: 8 total, 8 up, 8 in 2024-09-10T19:12:07.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:07.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:06 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:07.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-85"}]: dispatch 2024-09-10T19:12:07.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:06 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-85"}]: dispatch 2024-09-10T19:12:07.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:06 smithi119 ceph-mon[29577]: osdmap e453: 8 total, 8 up, 8 in 2024-09-10T19:12:07.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:07.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:06 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:07.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-85"}]: dispatch 2024-09-10T19:12:07.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:06 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-85"}]: dispatch 2024-09-10T19:12:07.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:06 smithi119 ceph-mon[40674]: osdmap e453: 8 total, 8 up, 8 in 2024-09-10T19:12:07.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:07.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:06 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:07.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-85"}]: dispatch 2024-09-10T19:12:07.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:06 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-85"}]: dispatch 2024-09-10T19:12:08.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:07 smithi194 ceph-mon[32464]: osdmap e454: 8 total, 8 up, 8 in 2024-09-10T19:12:08.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:07 smithi194 ceph-mon[32464]: pgmap v691: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:08.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:07 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:08.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:07 smithi119 ceph-mon[29577]: osdmap e454: 8 total, 8 up, 8 in 2024-09-10T19:12:08.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:07 smithi119 ceph-mon[29577]: pgmap v691: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:08.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:07 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:08.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:07 smithi119 ceph-mon[40674]: osdmap e454: 8 total, 8 up, 8 in 2024-09-10T19:12:08.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:07 smithi119 ceph-mon[40674]: pgmap v691: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:08.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:07 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:08.839 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:12:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:12:09.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:08 smithi194 ceph-mon[32464]: osdmap e455: 8 total, 8 up, 8 in 2024-09-10T19:12:09.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:09.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:09.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:08 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:09.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:08 smithi119 ceph-mon[29577]: osdmap e455: 8 total, 8 up, 8 in 2024-09-10T19:12:09.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:09.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:09.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:08 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:09.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:08 smithi119 ceph-mon[40674]: osdmap e455: 8 total, 8 up, 8 in 2024-09-10T19:12:09.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:09.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:09.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:08 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:10.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:10.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:09 smithi194 ceph-mon[32464]: osdmap e456: 8 total, 8 up, 8 in 2024-09-10T19:12:10.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:10.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:09 smithi194 ceph-mon[32464]: pgmap v694: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-09-10T19:12:10.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:10.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:09 smithi119 ceph-mon[29577]: osdmap e456: 8 total, 8 up, 8 in 2024-09-10T19:12:10.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:10.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:09 smithi119 ceph-mon[29577]: pgmap v694: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-09-10T19:12:10.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:10.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:09 smithi119 ceph-mon[40674]: osdmap e456: 8 total, 8 up, 8 in 2024-09-10T19:12:10.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:10.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:09 smithi119 ceph-mon[40674]: pgmap v694: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-09-10T19:12:10.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:12:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:12:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:11.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:10 smithi194 ceph-mon[32464]: osdmap e457: 8 total, 8 up, 8 in 2024-09-10T19:12:11.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:11.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:10 smithi119 ceph-mon[29577]: osdmap e457: 8 total, 8 up, 8 in 2024-09-10T19:12:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:11.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:10 smithi119 ceph-mon[40674]: osdmap e457: 8 total, 8 up, 8 in 2024-09-10T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:11.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:12.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:12:12.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:11 smithi194 ceph-mon[32464]: osdmap e458: 8 total, 8 up, 8 in 2024-09-10T19:12:12.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_tier","val": "test-rados-api-smithi119-79782-89-test-flush"}]: dispatch 2024-09-10T19:12:12.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_tier","val": "test-rados-api-smithi119-79782-89-test-flush"}]: dispatch 2024-09-10T19:12:12.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:11 smithi194 ceph-mon[32464]: pgmap v697: 260 pgs: 32 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-09-10T19:12:12.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_tier","val": "test-rados-api-smithi119-79782-89-test-flush"}]': finished 2024-09-10T19:12:12.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:11 smithi194 ceph-mon[32464]: osdmap e459: 8 total, 8 up, 8 in 2024-09-10T19:12:12.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:12:12.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:11 smithi119 ceph-mon[29577]: osdmap e458: 8 total, 8 up, 8 in 2024-09-10T19:12:12.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_tier","val": "test-rados-api-smithi119-79782-89-test-flush"}]: dispatch 2024-09-10T19:12:12.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_tier","val": "test-rados-api-smithi119-79782-89-test-flush"}]: dispatch 2024-09-10T19:12:12.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:11 smithi119 ceph-mon[29577]: pgmap v697: 260 pgs: 32 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-09-10T19:12:12.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_tier","val": "test-rados-api-smithi119-79782-89-test-flush"}]': finished 2024-09-10T19:12:12.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:11 smithi119 ceph-mon[29577]: osdmap e459: 8 total, 8 up, 8 in 2024-09-10T19:12:12.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:12:12.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:11 smithi119 ceph-mon[40674]: osdmap e458: 8 total, 8 up, 8 in 2024-09-10T19:12:12.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_tier","val": "test-rados-api-smithi119-79782-89-test-flush"}]: dispatch 2024-09-10T19:12:12.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_tier","val": "test-rados-api-smithi119-79782-89-test-flush"}]: dispatch 2024-09-10T19:12:12.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:11 smithi119 ceph-mon[40674]: pgmap v697: 260 pgs: 32 unknown, 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2024-09-10T19:12:12.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_tier","val": "test-rados-api-smithi119-79782-89-test-flush"}]': finished 2024-09-10T19:12:12.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:11 smithi119 ceph-mon[40674]: osdmap e459: 8 total, 8 up, 8 in 2024-09-10T19:12:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:12:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:12:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:12 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:13.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:12:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:12:12] "GET /metrics HTTP/1.1" 200 228518 "" "Prometheus/2.43.0" 2024-09-10T19:12:13.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:12:13.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:12:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:12 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:13.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:12:13.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:12:13.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:12 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:14.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:12:14.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:13 smithi194 ceph-mon[32464]: osdmap e460: 8 total, 8 up, 8 in 2024-09-10T19:12:14.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:12:14.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:12:14.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:13 smithi194 ceph-mon[32464]: pgmap v700: 260 pgs: 260 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:14.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:12:14.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:13 smithi194 ceph-mon[32464]: osdmap e461: 8 total, 8 up, 8 in 2024-09-10T19:12:14.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:12:14.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:13 smithi119 ceph-mon[29577]: osdmap e460: 8 total, 8 up, 8 in 2024-09-10T19:12:14.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:12:14.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:12:14.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:13 smithi119 ceph-mon[29577]: pgmap v700: 260 pgs: 260 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:14.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:12:14.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:13 smithi119 ceph-mon[29577]: osdmap e461: 8 total, 8 up, 8 in 2024-09-10T19:12:14.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:12:14.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:13 smithi119 ceph-mon[40674]: osdmap e460: 8 total, 8 up, 8 in 2024-09-10T19:12:14.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:12:14.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:12:14.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:13 smithi119 ceph-mon[40674]: pgmap v700: 260 pgs: 260 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:14.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:12:14.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:13 smithi119 ceph-mon[40674]: osdmap e461: 8 total, 8 up, 8 in 2024-09-10T19:12:16.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:15 smithi194 ceph-mon[32464]: osdmap e462: 8 total, 8 up, 8 in 2024-09-10T19:12:16.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:16.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:16.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-87"}]: dispatch 2024-09-10T19:12:16.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-87"}]: dispatch 2024-09-10T19:12:16.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:15 smithi194 ceph-mon[32464]: pgmap v703: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:16.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:15 smithi119 ceph-mon[29577]: osdmap e462: 8 total, 8 up, 8 in 2024-09-10T19:12:16.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:16.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:16.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-87"}]: dispatch 2024-09-10T19:12:16.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-87"}]: dispatch 2024-09-10T19:12:16.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:15 smithi119 ceph-mon[29577]: pgmap v703: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:16.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:15 smithi119 ceph-mon[40674]: osdmap e462: 8 total, 8 up, 8 in 2024-09-10T19:12:16.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:16.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:16.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-87"}]: dispatch 2024-09-10T19:12:16.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-87"}]: dispatch 2024-09-10T19:12:16.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:15 smithi119 ceph-mon[40674]: pgmap v703: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:17.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:16 smithi194 ceph-mon[32464]: osdmap e463: 8 total, 8 up, 8 in 2024-09-10T19:12:17.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:16 smithi119 ceph-mon[29577]: osdmap e463: 8 total, 8 up, 8 in 2024-09-10T19:12:17.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:16 smithi119 ceph-mon[40674]: osdmap e463: 8 total, 8 up, 8 in 2024-09-10T19:12:18.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:17 smithi194 ceph-mon[32464]: osdmap e464: 8 total, 8 up, 8 in 2024-09-10T19:12:18.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:17 smithi194 ceph-mon[32464]: pgmap v706: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:17 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:17 smithi119 ceph-mon[29577]: osdmap e464: 8 total, 8 up, 8 in 2024-09-10T19:12:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:17 smithi119 ceph-mon[29577]: pgmap v706: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:18.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:17 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:18.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:17 smithi119 ceph-mon[40674]: osdmap e464: 8 total, 8 up, 8 in 2024-09-10T19:12:18.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:18.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:18.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:18.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:17 smithi119 ceph-mon[40674]: pgmap v706: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:18.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:17 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:18.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:12:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:12:19.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:18 smithi194 ceph-mon[32464]: osdmap e465: 8 total, 8 up, 8 in 2024-09-10T19:12:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:19.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:18 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:18 smithi119 ceph-mon[29577]: osdmap e465: 8 total, 8 up, 8 in 2024-09-10T19:12:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:19.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:18 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:19.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:18 smithi119 ceph-mon[40674]: osdmap e465: 8 total, 8 up, 8 in 2024-09-10T19:12:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:12:19.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:18 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:20.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:12:20.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:19 smithi119 ceph-mon[29577]: osdmap e466: 8 total, 8 up, 8 in 2024-09-10T19:12:20.218 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:19 smithi119 ceph-mon[29577]: pgmap v709: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:12:20.219 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:12:20.219 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:19 smithi119 ceph-mon[40674]: osdmap e466: 8 total, 8 up, 8 in 2024-09-10T19:12:20.219 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:19 smithi119 ceph-mon[40674]: pgmap v709: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:12:20.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:12:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:19 smithi194 ceph-mon[32464]: osdmap e466: 8 total, 8 up, 8 in 2024-09-10T19:12:20.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:19 smithi194 ceph-mon[32464]: pgmap v709: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:12:20.269 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:12:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:12:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:21.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:20 smithi194 ceph-mon[32464]: osdmap e467: 8 total, 8 up, 8 in 2024-09-10T19:12:21.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:20 smithi119 ceph-mon[29577]: osdmap e467: 8 total, 8 up, 8 in 2024-09-10T19:12:21.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:20 smithi119 ceph-mon[40674]: osdmap e467: 8 total, 8 up, 8 in 2024-09-10T19:12:22.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:21 smithi194 ceph-mon[32464]: osdmap e468: 8 total, 8 up, 8 in 2024-09-10T19:12:22.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:22.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:21 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:22.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-90"}]: dispatch 2024-09-10T19:12:22.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:21 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-90"}]: dispatch 2024-09-10T19:12:22.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:21 smithi194 ceph-mon[32464]: pgmap v712: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:12:22.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:21 smithi119 ceph-mon[29577]: osdmap e468: 8 total, 8 up, 8 in 2024-09-10T19:12:22.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:22.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:21 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:22.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-90"}]: dispatch 2024-09-10T19:12:22.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:21 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-90"}]: dispatch 2024-09-10T19:12:22.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:21 smithi119 ceph-mon[29577]: pgmap v712: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:12:22.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:21 smithi119 ceph-mon[40674]: osdmap e468: 8 total, 8 up, 8 in 2024-09-10T19:12:22.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:22.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:21 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:22.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-90"}]: dispatch 2024-09-10T19:12:22.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:21 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-90"}]: dispatch 2024-09-10T19:12:22.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:21 smithi119 ceph-mon[40674]: pgmap v712: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:12:23.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:22 smithi119 ceph-mon[29577]: osdmap e469: 8 total, 8 up, 8 in 2024-09-10T19:12:23.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:12:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:12:22] "GET /metrics HTTP/1.1" 200 228557 "" "Prometheus/2.43.0" 2024-09-10T19:12:23.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:22 smithi119 ceph-mon[40674]: osdmap e469: 8 total, 8 up, 8 in 2024-09-10T19:12:23.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:22 smithi194 ceph-mon[32464]: osdmap e469: 8 total, 8 up, 8 in 2024-09-10T19:12:24.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:23 smithi194 ceph-mon[32464]: osdmap e470: 8 total, 8 up, 8 in 2024-09-10T19:12:24.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:24.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:24.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:23 smithi194 ceph-mon[32464]: pgmap v715: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:12:24.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:23 smithi119 ceph-mon[29577]: osdmap e470: 8 total, 8 up, 8 in 2024-09-10T19:12:24.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:24.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:24.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:23 smithi119 ceph-mon[29577]: pgmap v715: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:12:24.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:23 smithi119 ceph-mon[40674]: osdmap e470: 8 total, 8 up, 8 in 2024-09-10T19:12:24.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:24.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:24.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:23 smithi119 ceph-mon[40674]: pgmap v715: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:12:25.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:24 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:25.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:25.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:24 smithi194 ceph-mon[32464]: osdmap e471: 8 total, 8 up, 8 in 2024-09-10T19:12:25.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:25.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:24 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:25.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:25.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:24 smithi119 ceph-mon[29577]: osdmap e471: 8 total, 8 up, 8 in 2024-09-10T19:12:25.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:25.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:24 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:25.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:25.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:24 smithi119 ceph-mon[40674]: osdmap e471: 8 total, 8 up, 8 in 2024-09-10T19:12:25.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:26.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:25 smithi194 ceph-mon[32464]: osdmap e472: 8 total, 8 up, 8 in 2024-09-10T19:12:26.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:25 smithi194 ceph-mon[32464]: pgmap v718: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:12:26.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:25 smithi119 ceph-mon[29577]: osdmap e472: 8 total, 8 up, 8 in 2024-09-10T19:12:26.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:25 smithi119 ceph-mon[29577]: pgmap v718: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:12:26.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:25 smithi119 ceph-mon[40674]: osdmap e472: 8 total, 8 up, 8 in 2024-09-10T19:12:26.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:25 smithi119 ceph-mon[40674]: pgmap v718: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-09-10T19:12:27.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:26 smithi194 ceph-mon[32464]: osdmap e473: 8 total, 8 up, 8 in 2024-09-10T19:12:27.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:27.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:27.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-92"}]: dispatch 2024-09-10T19:12:27.270 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-92"}]: dispatch 2024-09-10T19:12:27.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:26 smithi119 ceph-mon[29577]: osdmap e473: 8 total, 8 up, 8 in 2024-09-10T19:12:27.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:27.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:27.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-92"}]: dispatch 2024-09-10T19:12:27.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-92"}]: dispatch 2024-09-10T19:12:27.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:26 smithi119 ceph-mon[40674]: osdmap e473: 8 total, 8 up, 8 in 2024-09-10T19:12:27.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:27.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:27.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-92"}]: dispatch 2024-09-10T19:12:27.319 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-92"}]: dispatch 2024-09-10T19:12:28.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:27 smithi194 ceph-mon[32464]: osdmap e474: 8 total, 8 up, 8 in 2024-09-10T19:12:28.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:27 smithi194 ceph-mon[32464]: pgmap v721: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:27 smithi119 ceph-mon[29577]: osdmap e474: 8 total, 8 up, 8 in 2024-09-10T19:12:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:27 smithi119 ceph-mon[29577]: pgmap v721: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:28.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:27 smithi119 ceph-mon[40674]: osdmap e474: 8 total, 8 up, 8 in 2024-09-10T19:12:28.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:27 smithi119 ceph-mon[40674]: pgmap v721: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:28.985 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:12:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:12:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:28 smithi194 ceph-mon[32464]: osdmap e475: 8 total, 8 up, 8 in 2024-09-10T19:12:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:28 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:28 smithi119 ceph-mon[29577]: osdmap e475: 8 total, 8 up, 8 in 2024-09-10T19:12:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:28 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:29.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:28 smithi119 ceph-mon[40674]: osdmap e475: 8 total, 8 up, 8 in 2024-09-10T19:12:29.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:28 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:30.268 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:12:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:12:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:30.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:30.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:29 smithi194 ceph-mon[32464]: osdmap e476: 8 total, 8 up, 8 in 2024-09-10T19:12:30.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:30.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:29 smithi194 ceph-mon[32464]: pgmap v724: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:12:30.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:30.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:29 smithi119 ceph-mon[29577]: osdmap e476: 8 total, 8 up, 8 in 2024-09-10T19:12:30.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:30.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:29 smithi119 ceph-mon[29577]: pgmap v724: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:12:30.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:30.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:29 smithi119 ceph-mon[40674]: osdmap e476: 8 total, 8 up, 8 in 2024-09-10T19:12:30.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:30.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:29 smithi119 ceph-mon[40674]: pgmap v724: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:12:31.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:30 smithi194 ceph-mon[32464]: osdmap e477: 8 total, 8 up, 8 in 2024-09-10T19:12:31.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:30 smithi119 ceph-mon[29577]: osdmap e477: 8 total, 8 up, 8 in 2024-09-10T19:12:31.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:30 smithi119 ceph-mon[40674]: osdmap e477: 8 total, 8 up, 8 in 2024-09-10T19:12:32.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:32 smithi194 ceph-mon[32464]: osdmap e478: 8 total, 8 up, 8 in 2024-09-10T19:12:32.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:32 smithi194 ceph-mon[32464]: pgmap v727: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:12:32.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:32 smithi119 ceph-mon[29577]: osdmap e478: 8 total, 8 up, 8 in 2024-09-10T19:12:32.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:32 smithi119 ceph-mon[29577]: pgmap v727: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:12:32.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:32 smithi119 ceph-mon[40674]: osdmap e478: 8 total, 8 up, 8 in 2024-09-10T19:12:32.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:32 smithi119 ceph-mon[40674]: pgmap v727: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-09-10T19:12:33.002 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:12:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:12:32] "GET /metrics HTTP/1.1" 200 228558 "" "Prometheus/2.43.0" 2024-09-10T19:12:33.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:33 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:33.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:33 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:33.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:33 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:33.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:33 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:33.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:33 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:33.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:33 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:34.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:34 smithi194 ceph-mon[32464]: pgmap v728: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-09-10T19:12:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:34 smithi119 ceph-mon[29577]: pgmap v728: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-09-10T19:12:34.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:34 smithi119 ceph-mon[40674]: pgmap v728: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 3.1 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2024-09-10T19:12:36.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:36 smithi119 ceph-mon[29577]: pgmap v729: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-09-10T19:12:36.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:36 smithi119 ceph-mon[40674]: pgmap v729: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-09-10T19:12:36.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:36 smithi194 ceph-mon[32464]: pgmap v729: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-09-10T19:12:38.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:38 smithi119 ceph-mon[29577]: pgmap v730: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T19:12:38.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:38 smithi119 ceph-mon[40674]: pgmap v730: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T19:12:38.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:38 smithi194 ceph-mon[32464]: pgmap v730: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T19:12:39.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:12:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:12:39.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:39 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:39.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:39 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:39.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:39 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:40.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:40 smithi194 ceph-mon[32464]: pgmap v731: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 887 B/s wr, 3 op/s 2024-09-10T19:12:40.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:12:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:12:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:40.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:40 smithi119 ceph-mon[29577]: pgmap v731: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 887 B/s wr, 3 op/s 2024-09-10T19:12:40.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:40 smithi119 ceph-mon[40674]: pgmap v731: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 887 B/s wr, 3 op/s 2024-09-10T19:12:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:42 smithi119 ceph-mon[29577]: pgmap v732: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 801 B/s wr, 3 op/s 2024-09-10T19:12:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:42 smithi119 ceph-mon[29577]: osdmap e479: 8 total, 8 up, 8 in 2024-09-10T19:12:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:42 smithi119 ceph-mon[40674]: pgmap v732: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 801 B/s wr, 3 op/s 2024-09-10T19:12:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:42 smithi119 ceph-mon[40674]: osdmap e479: 8 total, 8 up, 8 in 2024-09-10T19:12:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:42 smithi194 ceph-mon[32464]: pgmap v732: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 801 B/s wr, 3 op/s 2024-09-10T19:12:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:42 smithi194 ceph-mon[32464]: osdmap e479: 8 total, 8 up, 8 in 2024-09-10T19:12:43.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:12:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:12:42] "GET /metrics HTTP/1.1" 200 228558 "" "Prometheus/2.43.0" 2024-09-10T19:12:44.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:44 smithi119 ceph-mon[29577]: pgmap v734: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:44.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:44 smithi119 ceph-mon[40674]: pgmap v734: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:44.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:44 smithi194 ceph-mon[32464]: pgmap v734: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: pgmap v735: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:12:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:12:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:12:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:12:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:12:46.568 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:12:46.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: pgmap v735: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:46.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:12:46.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.569 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:12:46.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:12:46.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:12:46.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:12:46.570 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:12:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: pgmap v735: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:12:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:12:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:12:46.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:12:46.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:12:46.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:12:46.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:12:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:47 smithi119 ceph-mon[29577]: Adjusting osd_memory_target on smithi194 to 3248M 2024-09-10T19:12:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:47 smithi119 ceph-mon[29577]: Adjusting osd_memory_target on smithi119 to 3248M 2024-09-10T19:12:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:47.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:47 smithi119 ceph-mon[29577]: osdmap e480: 8 total, 8 up, 8 in 2024-09-10T19:12:47.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:47 smithi119 ceph-mon[40674]: Adjusting osd_memory_target on smithi194 to 3248M 2024-09-10T19:12:47.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:47 smithi119 ceph-mon[40674]: Adjusting osd_memory_target on smithi119 to 3248M 2024-09-10T19:12:47.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:47.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:47 smithi119 ceph-mon[40674]: osdmap e480: 8 total, 8 up, 8 in 2024-09-10T19:12:47.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:47 smithi194 ceph-mon[32464]: Adjusting osd_memory_target on smithi194 to 3248M 2024-09-10T19:12:47.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:47 smithi194 ceph-mon[32464]: Adjusting osd_memory_target on smithi119 to 3248M 2024-09-10T19:12:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:47 smithi194 ceph-mon[32464]: osdmap e480: 8 total, 8 up, 8 in 2024-09-10T19:12:48.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:48 smithi119 ceph-mon[29577]: pgmap v736: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:48.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:48 smithi119 ceph-mon[40674]: pgmap v736: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:48.607 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:48 smithi194 ceph-mon[32464]: pgmap v736: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:49.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:12:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:12:49.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:49 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:49.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:49 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:49.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:49 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:50.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:50 smithi194 ceph-mon[32464]: pgmap v738: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:50.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:12:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:12:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:50.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:50 smithi119 ceph-mon[29577]: pgmap v738: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:50.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:50 smithi119 ceph-mon[40674]: pgmap v738: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:52.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:52 smithi119 ceph-mon[29577]: pgmap v739: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:52.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:52.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:52.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-94"}]: dispatch 2024-09-10T19:12:52.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:52 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-94"}]: dispatch 2024-09-10T19:12:52.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:52 smithi119 ceph-mon[40674]: pgmap v739: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:52.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:52.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:52.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-94"}]: dispatch 2024-09-10T19:12:52.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:52 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-94"}]: dispatch 2024-09-10T19:12:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:52 smithi194 ceph-mon[32464]: pgmap v739: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:12:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:12:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-94"}]: dispatch 2024-09-10T19:12:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:52 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-94"}]: dispatch 2024-09-10T19:12:53.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:12:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:12:52] "GET /metrics HTTP/1.1" 200 228531 "" "Prometheus/2.43.0" 2024-09-10T19:12:53.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:53 smithi194 ceph-mon[32464]: osdmap e481: 8 total, 8 up, 8 in 2024-09-10T19:12:53.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:53 smithi119 ceph-mon[29577]: osdmap e481: 8 total, 8 up, 8 in 2024-09-10T19:12:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:53 smithi119 ceph-mon[40674]: osdmap e481: 8 total, 8 up, 8 in 2024-09-10T19:12:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:54 smithi194 ceph-mon[32464]: pgmap v741: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:54 smithi194 ceph-mon[32464]: osdmap e482: 8 total, 8 up, 8 in 2024-09-10T19:12:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:54 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:54 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:54.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:54 smithi119 ceph-mon[29577]: pgmap v741: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:54 smithi119 ceph-mon[29577]: osdmap e482: 8 total, 8 up, 8 in 2024-09-10T19:12:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:54 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:54 smithi119 ceph-mon[40674]: pgmap v741: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:54 smithi119 ceph-mon[40674]: osdmap e482: 8 total, 8 up, 8 in 2024-09-10T19:12:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:54 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:54 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:55.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:55 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:55.772 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:55 smithi194 ceph-mon[32464]: osdmap e483: 8 total, 8 up, 8 in 2024-09-10T19:12:55.772 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:55 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:55 smithi119 ceph-mon[29577]: osdmap e483: 8 total, 8 up, 8 in 2024-09-10T19:12:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:55.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:55 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:55.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:55 smithi119 ceph-mon[40674]: osdmap e483: 8 total, 8 up, 8 in 2024-09-10T19:12:55.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:12:56.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:56 smithi194 ceph-mon[32464]: pgmap v744: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:56.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:56 smithi194 ceph-mon[32464]: osdmap e484: 8 total, 8 up, 8 in 2024-09-10T19:12:56.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:56 smithi119 ceph-mon[29577]: pgmap v744: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:56 smithi119 ceph-mon[29577]: osdmap e484: 8 total, 8 up, 8 in 2024-09-10T19:12:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:56 smithi119 ceph-mon[40674]: pgmap v744: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:56 smithi119 ceph-mon[40674]: osdmap e484: 8 total, 8 up, 8 in 2024-09-10T19:12:58.619 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:58 smithi194 ceph-mon[32464]: pgmap v746: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 208 B/s wr, 1 op/s 2024-09-10T19:12:58.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:58 smithi119 ceph-mon[29577]: pgmap v746: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 208 B/s wr, 1 op/s 2024-09-10T19:12:58.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:58 smithi119 ceph-mon[40674]: pgmap v746: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 208 B/s wr, 1 op/s 2024-09-10T19:12:59.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:12:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:12:59.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:12:59 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:12:59 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:59.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:12:59 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:00.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:00 smithi194 ceph-mon[32464]: pgmap v747: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-09-10T19:13:00.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:13:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:13:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:00 smithi119 ceph-mon[29577]: pgmap v747: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-09-10T19:13:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:00 smithi119 ceph-mon[40674]: pgmap v747: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 853 B/s wr, 2 op/s 2024-09-10T19:13:02.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:02 smithi119 ceph-mon[29577]: pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 648 B/s rd, 648 B/s wr, 2 op/s 2024-09-10T19:13:02.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:02 smithi119 ceph-mon[40674]: pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 648 B/s rd, 648 B/s wr, 2 op/s 2024-09-10T19:13:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:02.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:02 smithi194 ceph-mon[32464]: pgmap v748: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 648 B/s rd, 648 B/s wr, 2 op/s 2024-09-10T19:13:02.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:03.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:13:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:13:02] "GET /metrics HTTP/1.1" 200 228537 "" "Prometheus/2.43.0" 2024-09-10T19:13:04.358 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:13:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=sqlstore.transactions t=2024-09-10T19:13:04.091048261Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T19:13:04.358 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:13:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=cleanup t=2024-09-10T19:13:04.092670257Z level=info msg="Completed cleanup jobs" duration=13.012415ms 2024-09-10T19:13:04.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:04 smithi194 ceph-mon[32464]: pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 575 B/s wr, 2 op/s 2024-09-10T19:13:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:04 smithi119 ceph-mon[29577]: pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 575 B/s wr, 2 op/s 2024-09-10T19:13:04.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:04 smithi119 ceph-mon[40674]: pgmap v749: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 575 B/s wr, 2 op/s 2024-09-10T19:13:05.768 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:13:05 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:13:05.363+0000 7f775cf83640 -1 snap_mapper.add_oid found existing snaps mapped on 63:0f87b9b2:test-rados-api-smithi119-79782-97::foo:31, removing 2024-09-10T19:13:05.817 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:13:05 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0[44998]: 2024-09-10T19:13:05.363+0000 7f953f4dc640 -1 snap_mapper.add_oid found existing snaps mapped on 63:0f87b9b2:test-rados-api-smithi119-79782-97::foo:31, removing 2024-09-10T19:13:05.817 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:13:05 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1[50994]: 2024-09-10T19:13:05.362+0000 7f7565b75640 -1 snap_mapper.add_oid found existing snaps mapped on 63:0f87b9b2:test-rados-api-smithi119-79782-97::foo:31, removing 2024-09-10T19:13:06.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:06 smithi194 ceph-mon[32464]: pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:13:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:06 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-96"}]: dispatch 2024-09-10T19:13:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:06 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-96"}]: dispatch 2024-09-10T19:13:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:06 smithi119 ceph-mon[29577]: pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:13:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:06.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:06 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-96"}]: dispatch 2024-09-10T19:13:06.820 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:06 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-96"}]: dispatch 2024-09-10T19:13:06.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:06 smithi119 ceph-mon[40674]: pgmap v750: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:13:06.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:06.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:06 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:06.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-96"}]: dispatch 2024-09-10T19:13:06.820 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:06 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-96"}]: dispatch 2024-09-10T19:13:07.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:07 smithi194 ceph-mon[32464]: osdmap e485: 8 total, 8 up, 8 in 2024-09-10T19:13:07.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:07 smithi119 ceph-mon[29577]: osdmap e485: 8 total, 8 up, 8 in 2024-09-10T19:13:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:07 smithi119 ceph-mon[40674]: osdmap e485: 8 total, 8 up, 8 in 2024-09-10T19:13:08.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:13:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:13:08.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:08 smithi194 ceph-mon[32464]: pgmap v752: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-09-10T19:13:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:08 smithi194 ceph-mon[32464]: osdmap e486: 8 total, 8 up, 8 in 2024-09-10T19:13:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:08.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:08 smithi119 ceph-mon[29577]: pgmap v752: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-09-10T19:13:08.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:08 smithi119 ceph-mon[29577]: osdmap e486: 8 total, 8 up, 8 in 2024-09-10T19:13:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:08 smithi119 ceph-mon[40674]: pgmap v752: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 204 B/s wr, 1 op/s 2024-09-10T19:13:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:08 smithi119 ceph-mon[40674]: osdmap e486: 8 total, 8 up, 8 in 2024-09-10T19:13:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:09.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:09 smithi194 ceph-mon[32464]: osdmap e487: 8 total, 8 up, 8 in 2024-09-10T19:13:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:13:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:13:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:09 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:09 smithi119 ceph-mon[29577]: osdmap e487: 8 total, 8 up, 8 in 2024-09-10T19:13:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:13:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:13:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:09 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:09 smithi119 ceph-mon[40674]: osdmap e487: 8 total, 8 up, 8 in 2024-09-10T19:13:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:13:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:13:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:09 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:10.405 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:13:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:13:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:10.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:10 smithi194 ceph-mon[32464]: pgmap v755: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:13:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:10 smithi194 ceph-mon[32464]: osdmap e488: 8 total, 8 up, 8 in 2024-09-10T19:13:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-98", "mode": "writeback"}]: dispatch 2024-09-10T19:13:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-98", "mode": "writeback"}]: dispatch 2024-09-10T19:13:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:10 smithi119 ceph-mon[29577]: pgmap v755: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:13:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:10 smithi119 ceph-mon[29577]: osdmap e488: 8 total, 8 up, 8 in 2024-09-10T19:13:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-98", "mode": "writeback"}]: dispatch 2024-09-10T19:13:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-98", "mode": "writeback"}]: dispatch 2024-09-10T19:13:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:10 smithi119 ceph-mon[40674]: pgmap v755: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:13:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:13:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:10 smithi119 ceph-mon[40674]: osdmap e488: 8 total, 8 up, 8 in 2024-09-10T19:13:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-98", "mode": "writeback"}]: dispatch 2024-09-10T19:13:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-98", "mode": "writeback"}]: dispatch 2024-09-10T19:13:11.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:11 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:13:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-98", "mode": "writeback"}]': finished 2024-09-10T19:13:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:11 smithi194 ceph-mon[32464]: osdmap e489: 8 total, 8 up, 8 in 2024-09-10T19:13:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-98"}]: dispatch 2024-09-10T19:13:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-98"}]: dispatch 2024-09-10T19:13:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-98"}]': finished 2024-09-10T19:13:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:11 smithi194 ceph-mon[32464]: osdmap e490: 8 total, 8 up, 8 in 2024-09-10T19:13:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:13:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:13:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-98", "mode": "writeback"}]': finished 2024-09-10T19:13:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[40674]: osdmap e489: 8 total, 8 up, 8 in 2024-09-10T19:13:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-98"}]: dispatch 2024-09-10T19:13:11.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-98"}]: dispatch 2024-09-10T19:13:11.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-98"}]': finished 2024-09-10T19:13:11.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[40674]: osdmap e490: 8 total, 8 up, 8 in 2024-09-10T19:13:11.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:13:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:13:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-98", "mode": "writeback"}]': finished 2024-09-10T19:13:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[29577]: osdmap e489: 8 total, 8 up, 8 in 2024-09-10T19:13:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-98"}]: dispatch 2024-09-10T19:13:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-98"}]: dispatch 2024-09-10T19:13:11.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-98"}]': finished 2024-09-10T19:13:11.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[29577]: osdmap e490: 8 total, 8 up, 8 in 2024-09-10T19:13:11.819 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:13:12.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-mon[29577]: pgmap v758: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:13:12.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:13:12.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:13:12.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:13:12.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-mon[29577]: osdmap e491: 8 total, 8 up, 8 in 2024-09-10T19:13:12.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_count","val": "1"}]: dispatch 2024-09-10T19:13:12.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_count","val": "1"}]: dispatch 2024-09-10T19:13:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-mon[40674]: pgmap v758: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:13:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:13:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:13:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:13:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-mon[40674]: osdmap e491: 8 total, 8 up, 8 in 2024-09-10T19:13:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_count","val": "1"}]: dispatch 2024-09-10T19:13:12.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_count","val": "1"}]: dispatch 2024-09-10T19:13:12.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:12 smithi194 ceph-mon[32464]: pgmap v758: 228 pgs: 32 creating+peering, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:13:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:13:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:12 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:13:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:13:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:12 smithi194 ceph-mon[32464]: osdmap e491: 8 total, 8 up, 8 in 2024-09-10T19:13:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_count","val": "1"}]: dispatch 2024-09-10T19:13:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_count","val": "1"}]: dispatch 2024-09-10T19:13:13.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:13:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:13:12] "GET /metrics HTTP/1.1" 200 228537 "" "Prometheus/2.43.0" 2024-09-10T19:13:14.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:14 smithi194 ceph-mon[32464]: pgmap v761: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:13:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_count","val": "1"}]': finished 2024-09-10T19:13:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:14 smithi194 ceph-mon[32464]: osdmap e492: 8 total, 8 up, 8 in 2024-09-10T19:13:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:13:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:13:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:14 smithi119 ceph-mon[29577]: pgmap v761: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:13:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_count","val": "1"}]': finished 2024-09-10T19:13:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:14 smithi119 ceph-mon[29577]: osdmap e492: 8 total, 8 up, 8 in 2024-09-10T19:13:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:13:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:13:14.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:14 smithi119 ceph-mon[40674]: pgmap v761: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:13:14.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_count","val": "1"}]': finished 2024-09-10T19:13:14.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:14 smithi119 ceph-mon[40674]: osdmap e492: 8 total, 8 up, 8 in 2024-09-10T19:13:14.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:13:14.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:13:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:13:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:15 smithi194 ceph-mon[32464]: osdmap e493: 8 total, 8 up, 8 in 2024-09-10T19:13:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "target_max_objects","val": "250"}]: dispatch 2024-09-10T19:13:15.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "target_max_objects","val": "250"}]: dispatch 2024-09-10T19:13:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:13:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:15 smithi119 ceph-mon[29577]: osdmap e493: 8 total, 8 up, 8 in 2024-09-10T19:13:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "target_max_objects","val": "250"}]: dispatch 2024-09-10T19:13:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "target_max_objects","val": "250"}]: dispatch 2024-09-10T19:13:15.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:13:15.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:15 smithi119 ceph-mon[40674]: osdmap e493: 8 total, 8 up, 8 in 2024-09-10T19:13:15.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "target_max_objects","val": "250"}]: dispatch 2024-09-10T19:13:15.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "target_max_objects","val": "250"}]: dispatch 2024-09-10T19:13:16.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:16 smithi194 ceph-mon[32464]: pgmap v764: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:13:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "target_max_objects","val": "250"}]': finished 2024-09-10T19:13:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:16 smithi194 ceph-mon[32464]: osdmap e494: 8 total, 8 up, 8 in 2024-09-10T19:13:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:16 smithi119 ceph-mon[29577]: pgmap v764: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:13:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "target_max_objects","val": "250"}]': finished 2024-09-10T19:13:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:16 smithi119 ceph-mon[29577]: osdmap e494: 8 total, 8 up, 8 in 2024-09-10T19:13:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:16 smithi119 ceph-mon[40674]: pgmap v764: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:13:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-98","var": "target_max_objects","val": "250"}]': finished 2024-09-10T19:13:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:16 smithi119 ceph-mon[40674]: osdmap e494: 8 total, 8 up, 8 in 2024-09-10T19:13:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:16.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:13:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:17 smithi194 ceph-mon[32464]: osdmap e495: 8 total, 8 up, 8 in 2024-09-10T19:13:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98"}]: dispatch 2024-09-10T19:13:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98"}]: dispatch 2024-09-10T19:13:17.770 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:17.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:13:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:17 smithi119 ceph-mon[29577]: osdmap e495: 8 total, 8 up, 8 in 2024-09-10T19:13:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98"}]: dispatch 2024-09-10T19:13:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98"}]: dispatch 2024-09-10T19:13:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:17.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:13:17.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:17 smithi119 ceph-mon[40674]: osdmap e495: 8 total, 8 up, 8 in 2024-09-10T19:13:17.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98"}]: dispatch 2024-09-10T19:13:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98"}]: dispatch 2024-09-10T19:13:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:19.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:13:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:13:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:18 smithi194 ceph-mon[32464]: pgmap v767: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98"}]': finished 2024-09-10T19:13:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:18 smithi194 ceph-mon[32464]: osdmap e496: 8 total, 8 up, 8 in 2024-09-10T19:13:19.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:18 smithi119 ceph-mon[29577]: pgmap v767: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:19.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98"}]': finished 2024-09-10T19:13:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:18 smithi119 ceph-mon[29577]: osdmap e496: 8 total, 8 up, 8 in 2024-09-10T19:13:19.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:18 smithi119 ceph-mon[40674]: pgmap v767: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:19.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-98"}]': finished 2024-09-10T19:13:19.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:18 smithi119 ceph-mon[40674]: osdmap e496: 8 total, 8 up, 8 in 2024-09-10T19:13:20.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:19 smithi194 ceph-mon[32464]: osdmap e497: 8 total, 8 up, 8 in 2024-09-10T19:13:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:19 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:19 smithi194 ceph-mon[32464]: pgmap v770: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:19 smithi194 ceph-mon[32464]: osdmap e498: 8 total, 8 up, 8 in 2024-09-10T19:13:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:19 smithi119 ceph-mon[29577]: osdmap e497: 8 total, 8 up, 8 in 2024-09-10T19:13:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:19 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:19 smithi119 ceph-mon[29577]: pgmap v770: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:19 smithi119 ceph-mon[29577]: osdmap e498: 8 total, 8 up, 8 in 2024-09-10T19:13:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:20.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:19 smithi119 ceph-mon[40674]: osdmap e497: 8 total, 8 up, 8 in 2024-09-10T19:13:20.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:19 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:20.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:19 smithi119 ceph-mon[40674]: pgmap v770: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:19 smithi119 ceph-mon[40674]: osdmap e498: 8 total, 8 up, 8 in 2024-09-10T19:13:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:20.520 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:13:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:13:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:21.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:21 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:21 smithi194 ceph-mon[32464]: osdmap e499: 8 total, 8 up, 8 in 2024-09-10T19:13:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:13:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:21 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:13:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:21 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:21 smithi119 ceph-mon[29577]: osdmap e499: 8 total, 8 up, 8 in 2024-09-10T19:13:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:13:21.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:21 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:13:21.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:21 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:21.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:21 smithi119 ceph-mon[40674]: osdmap e499: 8 total, 8 up, 8 in 2024-09-10T19:13:21.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:13:21.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:21 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:13:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:22 smithi119 ceph-mon[29577]: pgmap v773: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:13:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:22 smithi119 ceph-mon[29577]: osdmap e500: 8 total, 8 up, 8 in 2024-09-10T19:13:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-100"}]: dispatch 2024-09-10T19:13:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:22 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-100"}]: dispatch 2024-09-10T19:13:22.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:22 smithi119 ceph-mon[40674]: pgmap v773: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:22.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:13:22.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:22 smithi119 ceph-mon[40674]: osdmap e500: 8 total, 8 up, 8 in 2024-09-10T19:13:22.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-100"}]: dispatch 2024-09-10T19:13:22.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:22 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-100"}]: dispatch 2024-09-10T19:13:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:22 smithi194 ceph-mon[32464]: pgmap v773: 228 pgs: 32 unknown, 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:13:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:22 smithi194 ceph-mon[32464]: osdmap e500: 8 total, 8 up, 8 in 2024-09-10T19:13:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-100"}]: dispatch 2024-09-10T19:13:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:22 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-100"}]: dispatch 2024-09-10T19:13:23.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:13:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:13:22] "GET /metrics HTTP/1.1" 200 228513 "" "Prometheus/2.43.0" 2024-09-10T19:13:23.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-100"}]': finished 2024-09-10T19:13:23.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:23 smithi194 ceph-mon[32464]: osdmap e501: 8 total, 8 up, 8 in 2024-09-10T19:13:23.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-100", "mode": "writeback"}]: dispatch 2024-09-10T19:13:23.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:23 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-100", "mode": "writeback"}]: dispatch 2024-09-10T19:13:23.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:23 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:13:23.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-100"}]': finished 2024-09-10T19:13:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:23 smithi119 ceph-mon[29577]: osdmap e501: 8 total, 8 up, 8 in 2024-09-10T19:13:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-100", "mode": "writeback"}]: dispatch 2024-09-10T19:13:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:23 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-100", "mode": "writeback"}]: dispatch 2024-09-10T19:13:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:23 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:13:23.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-6", "overlaypool": "test-rados-api-smithi119-79782-100"}]': finished 2024-09-10T19:13:23.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:23 smithi119 ceph-mon[40674]: osdmap e501: 8 total, 8 up, 8 in 2024-09-10T19:13:23.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-100", "mode": "writeback"}]: dispatch 2024-09-10T19:13:23.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:23 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-100", "mode": "writeback"}]: dispatch 2024-09-10T19:13:23.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:23 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:13:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:24 smithi194 ceph-mon[32464]: pgmap v776: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-100", "mode": "writeback"}]': finished 2024-09-10T19:13:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:24 smithi194 ceph-mon[32464]: osdmap e502: 8 total, 8 up, 8 in 2024-09-10T19:13:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:13:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:13:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:13:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:24 smithi194 ceph-mon[32464]: osdmap e503: 8 total, 8 up, 8 in 2024-09-10T19:13:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:13:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:13:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[29577]: pgmap v776: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-100", "mode": "writeback"}]': finished 2024-09-10T19:13:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[29577]: osdmap e502: 8 total, 8 up, 8 in 2024-09-10T19:13:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:13:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:13:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:13:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[29577]: osdmap e503: 8 total, 8 up, 8 in 2024-09-10T19:13:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:13:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:13:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[40674]: pgmap v776: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-100", "mode": "writeback"}]': finished 2024-09-10T19:13:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[40674]: osdmap e502: 8 total, 8 up, 8 in 2024-09-10T19:13:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:13:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:13:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:13:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[40674]: osdmap e503: 8 total, 8 up, 8 in 2024-09-10T19:13:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:13:24.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:13:26.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:26 smithi194 ceph-mon[32464]: pgmap v779: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:13:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:26 smithi194 ceph-mon[32464]: osdmap e504: 8 total, 8 up, 8 in 2024-09-10T19:13:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:13:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:13:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:26 smithi119 ceph-mon[29577]: pgmap v779: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:13:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:26 smithi119 ceph-mon[29577]: osdmap e504: 8 total, 8 up, 8 in 2024-09-10T19:13:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:13:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:13:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:26 smithi119 ceph-mon[40674]: pgmap v779: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:13:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:26 smithi119 ceph-mon[40674]: osdmap e504: 8 total, 8 up, 8 in 2024-09-10T19:13:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:13:26.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:13:27.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:27 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:13:27.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:27 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:13:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:27 smithi194 ceph-mon[32464]: osdmap e505: 8 total, 8 up, 8 in 2024-09-10T19:13:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-09-10T19:13:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:27 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-09-10T19:13:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:27 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:13:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:27 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:13:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:27 smithi119 ceph-mon[29577]: osdmap e505: 8 total, 8 up, 8 in 2024-09-10T19:13:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-09-10T19:13:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:27 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-09-10T19:13:27.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:27 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:13:27.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:27 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:13:27.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:27 smithi119 ceph-mon[40674]: osdmap e505: 8 total, 8 up, 8 in 2024-09-10T19:13:27.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-09-10T19:13:27.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:27 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-09-10T19:13:28.768 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:13:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:13:28.776 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:28 smithi194 ceph-mon[32464]: pgmap v782: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:28.776 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-09-10T19:13:28.776 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:28 smithi194 ceph-mon[32464]: osdmap e506: 8 total, 8 up, 8 in 2024-09-10T19:13:28.776 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:28.777 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:28 smithi119 ceph-mon[29577]: pgmap v782: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-09-10T19:13:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:28 smithi119 ceph-mon[29577]: osdmap e506: 8 total, 8 up, 8 in 2024-09-10T19:13:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:28 smithi119 ceph-mon[40674]: pgmap v782: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-09-10T19:13:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:28 smithi119 ceph-mon[40674]: osdmap e506: 8 total, 8 up, 8 in 2024-09-10T19:13:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:28.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:29.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:13:29.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:29 smithi194 ceph-mon[32464]: osdmap e507: 8 total, 8 up, 8 in 2024-09-10T19:13:29.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100"}]: dispatch 2024-09-10T19:13:29.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100"}]: dispatch 2024-09-10T19:13:29.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:29 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:29.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:29 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100"}]': finished 2024-09-10T19:13:29.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:29 smithi194 ceph-mon[32464]: osdmap e508: 8 total, 8 up, 8 in 2024-09-10T19:13:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:13:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:29 smithi119 ceph-mon[29577]: osdmap e507: 8 total, 8 up, 8 in 2024-09-10T19:13:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100"}]: dispatch 2024-09-10T19:13:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100"}]: dispatch 2024-09-10T19:13:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:29 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:29 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100"}]': finished 2024-09-10T19:13:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:29 smithi119 ceph-mon[29577]: osdmap e508: 8 total, 8 up, 8 in 2024-09-10T19:13:29.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]': finished 2024-09-10T19:13:29.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:29 smithi119 ceph-mon[40674]: osdmap e507: 8 total, 8 up, 8 in 2024-09-10T19:13:29.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100"}]: dispatch 2024-09-10T19:13:29.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100"}]: dispatch 2024-09-10T19:13:29.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:29 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:29.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:29 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-100"}]': finished 2024-09-10T19:13:29.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:29 smithi119 ceph-mon[40674]: osdmap e508: 8 total, 8 up, 8 in 2024-09-10T19:13:30.485 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:13:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:13:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:30.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:30 smithi194 ceph-mon[32464]: pgmap v785: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:13:30.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:30 smithi119 ceph-mon[29577]: pgmap v785: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:13:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:30 smithi119 ceph-mon[40674]: pgmap v785: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:13:31.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:31 smithi194 ceph-mon[32464]: osdmap e509: 8 total, 8 up, 8 in 2024-09-10T19:13:31.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:31 smithi119 ceph-mon[29577]: osdmap e509: 8 total, 8 up, 8 in 2024-09-10T19:13:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:31 smithi119 ceph-mon[40674]: osdmap e509: 8 total, 8 up, 8 in 2024-09-10T19:13:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:32 smithi194 ceph-mon[32464]: pgmap v788: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:13:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:32 smithi194 ceph-mon[32464]: osdmap e510: 8 total, 8 up, 8 in 2024-09-10T19:13:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:32.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:13:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:13:32] "GET /metrics HTTP/1.1" 200 228500 "" "Prometheus/2.43.0" 2024-09-10T19:13:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:32 smithi119 ceph-mon[29577]: pgmap v788: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:13:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:32 smithi119 ceph-mon[29577]: osdmap e510: 8 total, 8 up, 8 in 2024-09-10T19:13:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:32.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:32.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:32 smithi119 ceph-mon[40674]: pgmap v788: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:13:32.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:32 smithi119 ceph-mon[40674]: osdmap e510: 8 total, 8 up, 8 in 2024-09-10T19:13:32.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:32.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:32.819 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:33 smithi194 ceph-mon[32464]: osdmap e511: 8 total, 8 up, 8 in 2024-09-10T19:13:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:13:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:13:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:13:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:33 smithi119 ceph-mon[29577]: osdmap e511: 8 total, 8 up, 8 in 2024-09-10T19:13:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:13:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:13:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:13:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:33 smithi119 ceph-mon[40674]: osdmap e511: 8 total, 8 up, 8 in 2024-09-10T19:13:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-09-10T19:13:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:13:33.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-09-10T19:13:34.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:34 smithi194 ceph-mon[32464]: pgmap v791: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:13:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:34 smithi194 ceph-mon[32464]: osdmap e512: 8 total, 8 up, 8 in 2024-09-10T19:13:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:13:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:13:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:34 smithi119 ceph-mon[29577]: pgmap v791: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:13:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:34 smithi119 ceph-mon[29577]: osdmap e512: 8 total, 8 up, 8 in 2024-09-10T19:13:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:13:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:13:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:34 smithi119 ceph-mon[40674]: pgmap v791: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-09-10T19:13:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:34 smithi119 ceph-mon[40674]: osdmap e512: 8 total, 8 up, 8 in 2024-09-10T19:13:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:13:34.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-09-10T19:13:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:35 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:13:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:35 smithi119 ceph-mon[29577]: osdmap e513: 8 total, 8 up, 8 in 2024-09-10T19:13:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:13:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:35 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:13:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:35 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:13:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:35 smithi119 ceph-mon[40674]: osdmap e513: 8 total, 8 up, 8 in 2024-09-10T19:13:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:13:35.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:35 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:13:36.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:35 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-09-10T19:13:36.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:35 smithi194 ceph-mon[32464]: osdmap e513: 8 total, 8 up, 8 in 2024-09-10T19:13:36.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:13:36.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:35 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-09-10T19:13:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:36 smithi119 ceph-mon[29577]: pgmap v794: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:13:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:36 smithi119 ceph-mon[29577]: osdmap e514: 8 total, 8 up, 8 in 2024-09-10T19:13:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-102"}]: dispatch 2024-09-10T19:13:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-102"}]: dispatch 2024-09-10T19:13:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:36 smithi119 ceph-mon[40674]: pgmap v794: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:13:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:36 smithi119 ceph-mon[40674]: osdmap e514: 8 total, 8 up, 8 in 2024-09-10T19:13:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-102"}]: dispatch 2024-09-10T19:13:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-102"}]: dispatch 2024-09-10T19:13:37.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:36 smithi194 ceph-mon[32464]: pgmap v794: 228 pgs: 228 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:37.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-09-10T19:13:37.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:36 smithi194 ceph-mon[32464]: osdmap e514: 8 total, 8 up, 8 in 2024-09-10T19:13:37.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:37.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-6"}]: dispatch 2024-09-10T19:13:37.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1891525760' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-102"}]: dispatch 2024-09-10T19:13:37.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-6", "tierpool": "test-rados-api-smithi119-79782-102"}]: dispatch 2024-09-10T19:13:37.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:37 smithi119 ceph-mon[29577]: osdmap e515: 8 total, 8 up, 8 in 2024-09-10T19:13:37.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:37 smithi119 ceph-mon[40674]: osdmap e515: 8 total, 8 up, 8 in 2024-09-10T19:13:38.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:37 smithi194 ceph-mon[32464]: osdmap e515: 8 total, 8 up, 8 in 2024-09-10T19:13:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[29577]: pgmap v797: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[29577]: osdmap e516: 8 total, 8 up, 8 in 2024-09-10T19:13:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:13:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:13:38.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[40674]: pgmap v797: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[40674]: osdmap e516: 8 total, 8 up, 8 in 2024-09-10T19:13:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:13:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:38 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:13:39.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:13:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:13:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:38 smithi194 ceph-mon[32464]: pgmap v797: 196 pgs: 196 active+clean; 8.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:38 smithi194 ceph-mon[32464]: osdmap e516: 8 total, 8 up, 8 in 2024-09-10T19:13:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:13:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:38 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:13:39.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:13:39.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:39 smithi119 ceph-mon[29577]: osdmap e517: 8 total, 8 up, 8 in 2024-09-10T19:13:39.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:39 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi119-79782-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:39.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:39 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi119-79782-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:39.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:39 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:39.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:13:39.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:39 smithi119 ceph-mon[40674]: osdmap e517: 8 total, 8 up, 8 in 2024-09-10T19:13:39.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:39 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi119-79782-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:39.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:39 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi119-79782-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:39.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:39 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:40.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:13:40.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:39 smithi194 ceph-mon[32464]: osdmap e517: 8 total, 8 up, 8 in 2024-09-10T19:13:40.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:39 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi119-79782-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:40.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:39 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi119-79782-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:13:40.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:39 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:40.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:13:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:13:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:40 smithi119 ceph-mon[29577]: pgmap v800: 164 pgs: 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:40 smithi119 ceph-mon[29577]: osdmap e518: 8 total, 8 up, 8 in 2024-09-10T19:13:40.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:40 smithi119 ceph-mon[40674]: pgmap v800: 164 pgs: 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:40.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:40 smithi119 ceph-mon[40674]: osdmap e518: 8 total, 8 up, 8 in 2024-09-10T19:13:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:40 smithi194 ceph-mon[32464]: pgmap v800: 164 pgs: 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:40 smithi194 ceph-mon[32464]: osdmap e518: 8 total, 8 up, 8 in 2024-09-10T19:13:42.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:41 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi119-79782-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi119-79782-104"}]': finished 2024-09-10T19:13:42.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:41 smithi194 ceph-mon[32464]: osdmap e519: 8 total, 8 up, 8 in 2024-09-10T19:13:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:41 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi119-79782-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi119-79782-104"}]': finished 2024-09-10T19:13:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:41 smithi119 ceph-mon[29577]: osdmap e519: 8 total, 8 up, 8 in 2024-09-10T19:13:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:41 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi119-79782-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi119-79782-104"}]': finished 2024-09-10T19:13:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:41 smithi119 ceph-mon[40674]: osdmap e519: 8 total, 8 up, 8 in 2024-09-10T19:13:43.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:42 smithi194 ceph-mon[32464]: pgmap v803: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:42 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:13:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:42 smithi194 ceph-mon[32464]: osdmap e520: 8 total, 8 up, 8 in 2024-09-10T19:13:43.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:13:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:13:42] "GET /metrics HTTP/1.1" 200 228500 "" "Prometheus/2.43.0" 2024-09-10T19:13:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:42 smithi119 ceph-mon[29577]: pgmap v803: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:42 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:13:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:42 smithi119 ceph-mon[29577]: osdmap e520: 8 total, 8 up, 8 in 2024-09-10T19:13:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:42 smithi119 ceph-mon[40674]: pgmap v803: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:42 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:13:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:42 smithi119 ceph-mon[40674]: osdmap e520: 8 total, 8 up, 8 in 2024-09-10T19:13:44.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:43 smithi194 ceph-mon[32464]: osdmap e521: 8 total, 8 up, 8 in 2024-09-10T19:13:44.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:44.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:43 smithi119 ceph-mon[29577]: osdmap e521: 8 total, 8 up, 8 in 2024-09-10T19:13:44.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:44.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:43 smithi119 ceph-mon[40674]: osdmap e521: 8 total, 8 up, 8 in 2024-09-10T19:13:44.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:45.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:44 smithi194 ceph-mon[32464]: pgmap v806: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:13:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:44 smithi194 ceph-mon[32464]: osdmap e522: 8 total, 8 up, 8 in 2024-09-10T19:13:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:44 smithi119 ceph-mon[29577]: pgmap v806: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:13:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:44 smithi119 ceph-mon[29577]: osdmap e522: 8 total, 8 up, 8 in 2024-09-10T19:13:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:44 smithi119 ceph-mon[40674]: pgmap v806: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:13:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:44 smithi119 ceph-mon[40674]: osdmap e522: 8 total, 8 up, 8 in 2024-09-10T19:13:46.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:45 smithi194 ceph-mon[32464]: osdmap e523: 8 total, 8 up, 8 in 2024-09-10T19:13:46.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:46.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:45 smithi119 ceph-mon[29577]: osdmap e523: 8 total, 8 up, 8 in 2024-09-10T19:13:46.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:46.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:45 smithi119 ceph-mon[40674]: osdmap e523: 8 total, 8 up, 8 in 2024-09-10T19:13:46.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:46.864 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:46 smithi119 ceph-mon[29577]: pgmap v809: 236 pgs: 32 unknown, 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:13:46.864 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:46.865 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:46 smithi119 ceph-mon[29577]: osdmap e524: 8 total, 8 up, 8 in 2024-09-10T19:13:46.865 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-107", "tierpool": "test-rados-api-smithi119-79782-107-cache"}]: dispatch 2024-09-10T19:13:46.865 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:46 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:13:46.865 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:46 smithi119 ceph-mon[40674]: pgmap v809: 236 pgs: 32 unknown, 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:13:46.865 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:46.866 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:46 smithi119 ceph-mon[40674]: osdmap e524: 8 total, 8 up, 8 in 2024-09-10T19:13:46.866 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-107", "tierpool": "test-rados-api-smithi119-79782-107-cache"}]: dispatch 2024-09-10T19:13:46.866 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:46 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:13:47.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:46 smithi194 ceph-mon[32464]: pgmap v809: 236 pgs: 32 unknown, 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:13:47.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:47.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:46 smithi194 ceph-mon[32464]: osdmap e524: 8 total, 8 up, 8 in 2024-09-10T19:13:47.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-107", "tierpool": "test-rados-api-smithi119-79782-107-cache"}]: dispatch 2024-09-10T19:13:47.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:46 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:13:47.900 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-107", "tierpool": "test-rados-api-smithi119-79782-107-cache"}]': finished 2024-09-10T19:13:47.901 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[29577]: osdmap e525: 8 total, 8 up, 8 in 2024-09-10T19:13:47.901 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-107", "overlaypool": "test-rados-api-smithi119-79782-107-cache"}]: dispatch 2024-09-10T19:13:47.901 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:47.901 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:13:47.901 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-107", "overlaypool": "test-rados-api-smithi119-79782-107-cache"}]': finished 2024-09-10T19:13:47.901 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[29577]: osdmap e526: 8 total, 8 up, 8 in 2024-09-10T19:13:47.901 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-107-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:13:47.901 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-107", "tierpool": "test-rados-api-smithi119-79782-107-cache"}]': finished 2024-09-10T19:13:47.902 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[40674]: osdmap e525: 8 total, 8 up, 8 in 2024-09-10T19:13:47.902 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-107", "overlaypool": "test-rados-api-smithi119-79782-107-cache"}]: dispatch 2024-09-10T19:13:47.902 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:47.902 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:13:47.902 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-107", "overlaypool": "test-rados-api-smithi119-79782-107-cache"}]': finished 2024-09-10T19:13:47.902 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[40674]: osdmap e526: 8 total, 8 up, 8 in 2024-09-10T19:13:47.902 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-107-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:13:48.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-107", "tierpool": "test-rados-api-smithi119-79782-107-cache"}]': finished 2024-09-10T19:13:48.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:47 smithi194 ceph-mon[32464]: osdmap e525: 8 total, 8 up, 8 in 2024-09-10T19:13:48.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-107", "overlaypool": "test-rados-api-smithi119-79782-107-cache"}]: dispatch 2024-09-10T19:13:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:47 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:13:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-107", "overlaypool": "test-rados-api-smithi119-79782-107-cache"}]': finished 2024-09-10T19:13:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:47 smithi194 ceph-mon[32464]: osdmap e526: 8 total, 8 up, 8 in 2024-09-10T19:13:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-107-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:13:49.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:13:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:13:49.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:48 smithi194 ceph-mon[32464]: pgmap v812: 236 pgs: 32 unknown, 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:49.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:48 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:13:49.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:48 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-107-cache", "mode": "writeback"}]': finished 2024-09-10T19:13:49.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:48 smithi194 ceph-mon[32464]: osdmap e527: 8 total, 8 up, 8 in 2024-09-10T19:13:49.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:48 smithi119 ceph-mon[29577]: pgmap v812: 236 pgs: 32 unknown, 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:49.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:48 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:13:49.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:48 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-107-cache", "mode": "writeback"}]': finished 2024-09-10T19:13:49.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:48 smithi119 ceph-mon[29577]: osdmap e527: 8 total, 8 up, 8 in 2024-09-10T19:13:49.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:48 smithi119 ceph-mon[40674]: pgmap v812: 236 pgs: 32 unknown, 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:49.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:48 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:13:49.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:48 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-107-cache", "mode": "writeback"}]': finished 2024-09-10T19:13:49.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:48 smithi119 ceph-mon[40674]: osdmap e527: 8 total, 8 up, 8 in 2024-09-10T19:13:49.913 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-107"}]: dispatch 2024-09-10T19:13:49.914 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:49 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:49.914 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:49 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:49.914 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:49 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:50.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-107"}]: dispatch 2024-09-10T19:13:50.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:49 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:50.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:49 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:50.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:49 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:50.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-107"}]: dispatch 2024-09-10T19:13:50.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:49 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:50.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:49 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:50.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:49 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:50.268 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:13:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:13:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:51.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:50 smithi194 ceph-mon[32464]: pgmap v815: 236 pgs: 236 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:51.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-107"}]': finished 2024-09-10T19:13:51.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:50 smithi194 ceph-mon[32464]: osdmap e528: 8 total, 8 up, 8 in 2024-09-10T19:13:51.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-107", "tierpool": "test-rados-api-smithi119-79782-107-cache"}]: dispatch 2024-09-10T19:13:51.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:50 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:51.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:50 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:51.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:50 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:13:51.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:50 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:13:51.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:50 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:51.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[29577]: pgmap v815: 236 pgs: 236 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:51.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-107"}]': finished 2024-09-10T19:13:51.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[29577]: osdmap e528: 8 total, 8 up, 8 in 2024-09-10T19:13:51.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-107", "tierpool": "test-rados-api-smithi119-79782-107-cache"}]: dispatch 2024-09-10T19:13:51.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:51.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:51.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:13:51.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:13:51.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:51.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[40674]: pgmap v815: 236 pgs: 236 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:51.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-107"}]': finished 2024-09-10T19:13:51.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[40674]: osdmap e528: 8 total, 8 up, 8 in 2024-09-10T19:13:51.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-107", "tierpool": "test-rados-api-smithi119-79782-107-cache"}]: dispatch 2024-09-10T19:13:51.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:51.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:51.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:13:51.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:13:51.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:50 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:13:51.968 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:51 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:13:51.968 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-107", "tierpool": "test-rados-api-smithi119-79782-107-cache"}]': finished 2024-09-10T19:13:51.968 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:51 smithi119 ceph-mon[29577]: osdmap e529: 8 total, 8 up, 8 in 2024-09-10T19:13:51.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:51 smithi119 ceph-mon[29577]: osdmap e530: 8 total, 8 up, 8 in 2024-09-10T19:13:51.969 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:51 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:13:51.969 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-107", "tierpool": "test-rados-api-smithi119-79782-107-cache"}]': finished 2024-09-10T19:13:51.969 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:51 smithi119 ceph-mon[40674]: osdmap e529: 8 total, 8 up, 8 in 2024-09-10T19:13:51.969 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:51 smithi119 ceph-mon[40674]: osdmap e530: 8 total, 8 up, 8 in 2024-09-10T19:13:52.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:51 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:13:52.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3040595392' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-107", "tierpool": "test-rados-api-smithi119-79782-107-cache"}]': finished 2024-09-10T19:13:52.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:51 smithi194 ceph-mon[32464]: osdmap e529: 8 total, 8 up, 8 in 2024-09-10T19:13:52.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:51 smithi194 ceph-mon[32464]: osdmap e530: 8 total, 8 up, 8 in 2024-09-10T19:13:53.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:52 smithi194 ceph-mon[32464]: pgmap v818: 236 pgs: 236 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:53.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:52 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:13:53.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:52 smithi194 ceph-mon[32464]: osdmap e531: 8 total, 8 up, 8 in 2024-09-10T19:13:53.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:13:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:13:52] "GET /metrics HTTP/1.1" 200 230968 "" "Prometheus/2.43.0" 2024-09-10T19:13:53.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:52 smithi119 ceph-mon[29577]: pgmap v818: 236 pgs: 236 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:53.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:52 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:13:53.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:52 smithi119 ceph-mon[29577]: osdmap e531: 8 total, 8 up, 8 in 2024-09-10T19:13:53.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:52 smithi119 ceph-mon[40674]: pgmap v818: 236 pgs: 236 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:53.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:52 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:13:53.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:52 smithi119 ceph-mon[40674]: osdmap e531: 8 total, 8 up, 8 in 2024-09-10T19:13:54.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:13:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:53 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:13:54.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:54.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:54.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:54.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:54.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:13:54.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:53 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:13:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:13:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:53 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:13:55.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:54 smithi194 ceph-mon[32464]: pgmap v821: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:55.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:54 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:13:55.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:54 smithi194 ceph-mon[32464]: osdmap e532: 8 total, 8 up, 8 in 2024-09-10T19:13:55.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:54 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:55.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:54 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:55.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:54 smithi119 ceph-mon[29577]: pgmap v821: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:55.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:54 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:13:55.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:54 smithi119 ceph-mon[29577]: osdmap e532: 8 total, 8 up, 8 in 2024-09-10T19:13:55.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:55.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:54 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:55.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:54 smithi119 ceph-mon[40674]: pgmap v821: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:55.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:54 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:13:55.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:54 smithi119 ceph-mon[40674]: osdmap e532: 8 total, 8 up, 8 in 2024-09-10T19:13:55.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:54 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:55.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:54 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:13:56.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:55 smithi194 ceph-mon[32464]: osdmap e533: 8 total, 8 up, 8 in 2024-09-10T19:13:56.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:55 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-109"}]': finished 2024-09-10T19:13:56.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:55 smithi194 ceph-mon[32464]: osdmap e534: 8 total, 8 up, 8 in 2024-09-10T19:13:56.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:55 smithi119 ceph-mon[29577]: osdmap e533: 8 total, 8 up, 8 in 2024-09-10T19:13:56.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:55 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-109"}]': finished 2024-09-10T19:13:56.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:55 smithi119 ceph-mon[29577]: osdmap e534: 8 total, 8 up, 8 in 2024-09-10T19:13:56.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:55 smithi119 ceph-mon[40674]: osdmap e533: 8 total, 8 up, 8 in 2024-09-10T19:13:56.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:55 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-109"}]': finished 2024-09-10T19:13:56.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:55 smithi119 ceph-mon[40674]: osdmap e534: 8 total, 8 up, 8 in 2024-09-10T19:13:57.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:56 smithi194 ceph-mon[32464]: pgmap v824: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:57.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:56 smithi194 ceph-mon[32464]: osdmap e535: 8 total, 8 up, 8 in 2024-09-10T19:13:57.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:57.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:56 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:57.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:56 smithi119 ceph-mon[29577]: pgmap v824: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:57.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:56 smithi119 ceph-mon[29577]: osdmap e535: 8 total, 8 up, 8 in 2024-09-10T19:13:57.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:57.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:56 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:57.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:56 smithi119 ceph-mon[40674]: pgmap v824: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:13:57.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:56 smithi119 ceph-mon[40674]: osdmap e535: 8 total, 8 up, 8 in 2024-09-10T19:13:57.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:57.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:56 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:13:59.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:13:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:13:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:58 smithi194 ceph-mon[32464]: pgmap v827: 212 pgs: 40 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:58 smithi194 ceph-mon[32464]: osdmap e536: 8 total, 8 up, 8 in 2024-09-10T19:13:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:13:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:58 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:13:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:58 smithi194 ceph-mon[32464]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:13:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:58 smithi119 ceph-mon[40674]: pgmap v827: 212 pgs: 40 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:58 smithi119 ceph-mon[40674]: osdmap e536: 8 total, 8 up, 8 in 2024-09-10T19:13:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:13:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:58 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:13:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:58 smithi119 ceph-mon[40674]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:13:59.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:58 smithi119 ceph-mon[29577]: pgmap v827: 212 pgs: 40 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:13:59.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:13:59.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:58 smithi119 ceph-mon[29577]: osdmap e536: 8 total, 8 up, 8 in 2024-09-10T19:13:59.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:13:59.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:58 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:13:59.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:58 smithi119 ceph-mon[29577]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:00.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:59 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]': finished 2024-09-10T19:14:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:59 smithi194 ceph-mon[32464]: osdmap e537: 8 total, 8 up, 8 in 2024-09-10T19:14:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:59 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-109", "overlaypool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:14:00.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:13:59 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-109", "overlaypool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:14:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:59 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]': finished 2024-09-10T19:14:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:59 smithi119 ceph-mon[40674]: osdmap e537: 8 total, 8 up, 8 in 2024-09-10T19:14:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:59 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-109", "overlaypool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:14:00.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:13:59 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-109", "overlaypool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:14:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:59 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]': finished 2024-09-10T19:14:00.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:59 smithi119 ceph-mon[29577]: osdmap e537: 8 total, 8 up, 8 in 2024-09-10T19:14:00.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:59 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-109", "overlaypool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:14:00.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:13:59 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-109", "overlaypool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:14:00.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:14:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:14:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:14:01.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:00 smithi194 ceph-mon[32464]: pgmap v830: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:01.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:00 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-109", "overlaypool": "test-rados-api-smithi119-79782-109-cache"}]': finished 2024-09-10T19:14:01.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:00 smithi194 ceph-mon[32464]: osdmap e538: 8 total, 8 up, 8 in 2024-09-10T19:14:01.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-109-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:14:01.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:00 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-109-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:14:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:00 smithi119 ceph-mon[29577]: pgmap v830: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:00 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-109", "overlaypool": "test-rados-api-smithi119-79782-109-cache"}]': finished 2024-09-10T19:14:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:00 smithi119 ceph-mon[29577]: osdmap e538: 8 total, 8 up, 8 in 2024-09-10T19:14:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-109-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:14:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:00 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-109-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:14:01.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:00 smithi119 ceph-mon[40674]: pgmap v830: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:01.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:00 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-109", "overlaypool": "test-rados-api-smithi119-79782-109-cache"}]': finished 2024-09-10T19:14:01.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:00 smithi119 ceph-mon[40674]: osdmap e538: 8 total, 8 up, 8 in 2024-09-10T19:14:01.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-109-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:14:01.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:00 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-109-cache", "mode": "writeback"}]: dispatch 2024-09-10T19:14:02.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:01 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:14:02.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:01 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-109-cache", "mode": "writeback"}]': finished 2024-09-10T19:14:02.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:01 smithi194 ceph-mon[32464]: osdmap e539: 8 total, 8 up, 8 in 2024-09-10T19:14:02.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:14:02.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:01 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:14:02.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:01 smithi194 ceph-mon[32464]: pgmap v833: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:01 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:14:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:01 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-109-cache", "mode": "writeback"}]': finished 2024-09-10T19:14:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:01 smithi119 ceph-mon[29577]: osdmap e539: 8 total, 8 up, 8 in 2024-09-10T19:14:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:14:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:01 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:14:02.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:01 smithi119 ceph-mon[29577]: pgmap v833: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:02.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:01 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:14:02.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:01 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-109-cache", "mode": "writeback"}]': finished 2024-09-10T19:14:02.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:01 smithi119 ceph-mon[40674]: osdmap e539: 8 total, 8 up, 8 in 2024-09-10T19:14:02.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:14:02.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:01 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:14:02.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:01 smithi119 ceph-mon[40674]: pgmap v833: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:03.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:02 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:14:03.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:02 smithi194 ceph-mon[32464]: osdmap e540: 8 total, 8 up, 8 in 2024-09-10T19:14:03.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:14:03.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:02 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:14:03.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:14:03.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:03.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:14:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:14:02] "GET /metrics HTTP/1.1" 200 230955 "" "Prometheus/2.43.0" 2024-09-10T19:14:03.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:02 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:14:03.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:02 smithi119 ceph-mon[29577]: osdmap e540: 8 total, 8 up, 8 in 2024-09-10T19:14:03.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:14:03.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:02 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:14:03.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:14:03.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:03.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:02 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:14:03.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:02 smithi119 ceph-mon[40674]: osdmap e540: 8 total, 8 up, 8 in 2024-09-10T19:14:03.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:14:03.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:02 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:14:03.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:14:03.070 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:04.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:14:04.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:03 smithi194 ceph-mon[32464]: osdmap e541: 8 total, 8 up, 8 in 2024-09-10T19:14:04.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:14:04.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:03 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:14:04.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:03 smithi194 ceph-mon[32464]: pgmap v836: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:04.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:14:04.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:03 smithi119 ceph-mon[29577]: osdmap e541: 8 total, 8 up, 8 in 2024-09-10T19:14:04.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:14:04.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:03 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:14:04.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:03 smithi119 ceph-mon[29577]: pgmap v836: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:04.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:14:04.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:03 smithi119 ceph-mon[40674]: osdmap e541: 8 total, 8 up, 8 in 2024-09-10T19:14:04.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:14:04.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:03 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:14:04.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:03 smithi119 ceph-mon[40674]: pgmap v836: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:04 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:14:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:14:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:04 smithi194 ceph-mon[32464]: osdmap e542: 8 total, 8 up, 8 in 2024-09-10T19:14:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-09-10T19:14:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:04 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-09-10T19:14:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:04 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:14:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:14:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:04 smithi119 ceph-mon[29577]: osdmap e542: 8 total, 8 up, 8 in 2024-09-10T19:14:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-09-10T19:14:05.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:04 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-09-10T19:14:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:04 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:14:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:14:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:04 smithi119 ceph-mon[40674]: osdmap e542: 8 total, 8 up, 8 in 2024-09-10T19:14:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-09-10T19:14:05.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:04 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-09-10T19:14:06.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-09-10T19:14:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:05 smithi194 ceph-mon[32464]: osdmap e543: 8 total, 8 up, 8 in 2024-09-10T19:14:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:05 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:06.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:05 smithi194 ceph-mon[32464]: pgmap v839: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-09-10T19:14:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:05 smithi119 ceph-mon[29577]: osdmap e543: 8 total, 8 up, 8 in 2024-09-10T19:14:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:05 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:06.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:05 smithi119 ceph-mon[29577]: pgmap v839: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:06.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-09-10T19:14:06.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:05 smithi119 ceph-mon[40674]: osdmap e543: 8 total, 8 up, 8 in 2024-09-10T19:14:06.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:06.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:05 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:06.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:05 smithi119 ceph-mon[40674]: pgmap v839: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:07.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:06 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-109"}]': finished 2024-09-10T19:14:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:06 smithi194 ceph-mon[32464]: osdmap e544: 8 total, 8 up, 8 in 2024-09-10T19:14:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:14:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:06 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:14:07.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:06 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-109"}]': finished 2024-09-10T19:14:07.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:06 smithi119 ceph-mon[29577]: osdmap e544: 8 total, 8 up, 8 in 2024-09-10T19:14:07.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:14:07.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:06 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:14:07.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:06 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-109"}]': finished 2024-09-10T19:14:07.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:06 smithi119 ceph-mon[40674]: osdmap e544: 8 total, 8 up, 8 in 2024-09-10T19:14:07.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:14:07.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:06 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]: dispatch 2024-09-10T19:14:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:07 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]': finished 2024-09-10T19:14:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:07 smithi194 ceph-mon[32464]: osdmap e545: 8 total, 8 up, 8 in 2024-09-10T19:14:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:07 smithi194 ceph-mon[32464]: pgmap v842: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:07 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:08.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:07 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]': finished 2024-09-10T19:14:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:07 smithi119 ceph-mon[29577]: osdmap e545: 8 total, 8 up, 8 in 2024-09-10T19:14:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:07 smithi119 ceph-mon[29577]: pgmap v842: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:08.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:07 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:08.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:07 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-109", "tierpool": "test-rados-api-smithi119-79782-109-cache"}]': finished 2024-09-10T19:14:08.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:07 smithi119 ceph-mon[40674]: osdmap e545: 8 total, 8 up, 8 in 2024-09-10T19:14:08.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:07 smithi119 ceph-mon[40674]: pgmap v842: 212 pgs: 212 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:08.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:07 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:09.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:08 smithi194 ceph-mon[32464]: osdmap e546: 8 total, 8 up, 8 in 2024-09-10T19:14:09.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:08 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:09.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:14:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:14:09.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:08 smithi119 ceph-mon[29577]: osdmap e546: 8 total, 8 up, 8 in 2024-09-10T19:14:09.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:08 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:09.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:08 smithi119 ceph-mon[40674]: osdmap e546: 8 total, 8 up, 8 in 2024-09-10T19:14:09.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:08 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:10.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:09 smithi194 ceph-mon[32464]: osdmap e547: 8 total, 8 up, 8 in 2024-09-10T19:14:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:09 smithi194 ceph-mon[32464]: pgmap v845: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:09 smithi119 ceph-mon[29577]: osdmap e547: 8 total, 8 up, 8 in 2024-09-10T19:14:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:10.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:09 smithi119 ceph-mon[29577]: pgmap v845: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:10.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:09 smithi119 ceph-mon[40674]: osdmap e547: 8 total, 8 up, 8 in 2024-09-10T19:14:10.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:10.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:10.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:09 smithi119 ceph-mon[40674]: pgmap v845: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:10.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:14:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:14:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:14:11.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]': finished 2024-09-10T19:14:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:10 smithi194 ceph-mon[32464]: osdmap e548: 8 total, 8 up, 8 in 2024-09-10T19:14:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:10 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:11.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]': finished 2024-09-10T19:14:11.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:10 smithi119 ceph-mon[29577]: osdmap e548: 8 total, 8 up, 8 in 2024-09-10T19:14:11.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:11.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:10 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:11.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-109"}]': finished 2024-09-10T19:14:11.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:10 smithi119 ceph-mon[40674]: osdmap e548: 8 total, 8 up, 8 in 2024-09-10T19:14:11.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3856169937' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:11.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:10 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]: dispatch 2024-09-10T19:14:12.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:11 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]': finished 2024-09-10T19:14:12.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:11 smithi194 ceph-mon[32464]: osdmap e549: 8 total, 8 up, 8 in 2024-09-10T19:14:12.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:11 smithi194 ceph-mon[32464]: pgmap v848: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:12.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:11 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]': finished 2024-09-10T19:14:12.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:11 smithi119 ceph-mon[29577]: osdmap e549: 8 total, 8 up, 8 in 2024-09-10T19:14:12.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:11 smithi119 ceph-mon[29577]: pgmap v848: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:12.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:11 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-109"}]': finished 2024-09-10T19:14:12.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:11 smithi119 ceph-mon[40674]: osdmap e549: 8 total, 8 up, 8 in 2024-09-10T19:14:12.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:11 smithi119 ceph-mon[40674]: pgmap v848: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:13.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:12 smithi194 ceph-mon[32464]: osdmap e550: 8 total, 8 up, 8 in 2024-09-10T19:14:13.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:14:13.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:12 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:14:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:12 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:13.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:14:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:14:12] "GET /metrics HTTP/1.1" 200 230955 "" "Prometheus/2.43.0" 2024-09-10T19:14:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:12 smithi119 ceph-mon[29577]: osdmap e550: 8 total, 8 up, 8 in 2024-09-10T19:14:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:14:13.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:12 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:14:13.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:12 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:12 smithi119 ceph-mon[40674]: osdmap e550: 8 total, 8 up, 8 in 2024-09-10T19:14:13.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:14:13.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:12 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:14:13.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:12 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:13.766 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]': finished 2024-09-10T19:14:13.766 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:13 smithi119 ceph-mon[29577]: osdmap e551: 8 total, 8 up, 8 in 2024-09-10T19:14:13.766 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:14:13.767 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:13 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:14:14.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]': finished 2024-09-10T19:14:14.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:13 smithi194 ceph-mon[32464]: osdmap e551: 8 total, 8 up, 8 in 2024-09-10T19:14:14.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:14:14.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:13 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:14:14.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:13 smithi194 ceph-mon[32464]: pgmap v851: 164 pgs: 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:14:14.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:13 smithi119 ceph-mon[29577]: pgmap v851: 164 pgs: 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:14:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi119-79782-104"}]': finished 2024-09-10T19:14:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:13 smithi119 ceph-mon[40674]: osdmap e551: 8 total, 8 up, 8 in 2024-09-10T19:14:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2738845549' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:14:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:13 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]: dispatch 2024-09-10T19:14:14.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:13 smithi119 ceph-mon[40674]: pgmap v851: 164 pgs: 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:14:15.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]': finished 2024-09-10T19:14:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:14 smithi194 ceph-mon[32464]: osdmap e552: 8 total, 8 up, 8 in 2024-09-10T19:14:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:14:15.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]': finished 2024-09-10T19:14:15.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:14 smithi119 ceph-mon[29577]: osdmap e552: 8 total, 8 up, 8 in 2024-09-10T19:14:15.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:15.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:15.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:14:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi119-79782-104"}]': finished 2024-09-10T19:14:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:14 smithi119 ceph-mon[40674]: osdmap e552: 8 total, 8 up, 8 in 2024-09-10T19:14:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-09-10T19:14:16.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:14:16.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:15 smithi119 ceph-mon[29577]: osdmap e553: 8 total, 8 up, 8 in 2024-09-10T19:14:16.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:16.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:15 smithi119 ceph-mon[29577]: pgmap v854: 164 pgs: 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:14:16.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:14:16.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:15 smithi119 ceph-mon[40674]: osdmap e553: 8 total, 8 up, 8 in 2024-09-10T19:14:16.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:16.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:15 smithi119 ceph-mon[40674]: pgmap v854: 164 pgs: 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:14:16.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi119-79782-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T19:14:16.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:15 smithi194 ceph-mon[32464]: osdmap e553: 8 total, 8 up, 8 in 2024-09-10T19:14:16.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:16.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:15 smithi194 ceph-mon[32464]: pgmap v854: 164 pgs: 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:14:17.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:16 smithi119 ceph-mon[29577]: osdmap e554: 8 total, 8 up, 8 in 2024-09-10T19:14:17.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:16 smithi119 ceph-mon[40674]: osdmap e554: 8 total, 8 up, 8 in 2024-09-10T19:14:17.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:16 smithi194 ceph-mon[32464]: osdmap e554: 8 total, 8 up, 8 in 2024-09-10T19:14:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:14:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:17 smithi119 ceph-mon[29577]: osdmap e555: 8 total, 8 up, 8 in 2024-09-10T19:14:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:14:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:17 smithi119 ceph-mon[29577]: pgmap v857: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:17 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:18.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:14:18.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:17 smithi119 ceph-mon[40674]: osdmap e555: 8 total, 8 up, 8 in 2024-09-10T19:14:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:14:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:17 smithi119 ceph-mon[40674]: pgmap v857: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:18.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:17 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:18.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi119-79782-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:14:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:17 smithi194 ceph-mon[32464]: osdmap e555: 8 total, 8 up, 8 in 2024-09-10T19:14:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:14:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:17 smithi194 ceph-mon[32464]: pgmap v857: 172 pgs: 8 unknown, 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:18.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:17 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:19.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:18 smithi194 ceph-mon[32464]: osdmap e556: 8 total, 8 up, 8 in 2024-09-10T19:14:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:14:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:14:19.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:18 smithi194 ceph-mon[32464]: osdmap e557: 8 total, 8 up, 8 in 2024-09-10T19:14:19.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:14:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:14:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:18 smithi119 ceph-mon[29577]: osdmap e556: 8 total, 8 up, 8 in 2024-09-10T19:14:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:14:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:14:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:18 smithi119 ceph-mon[29577]: osdmap e557: 8 total, 8 up, 8 in 2024-09-10T19:14:19.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:18 smithi119 ceph-mon[40674]: osdmap e556: 8 total, 8 up, 8 in 2024-09-10T19:14:19.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:14:19.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:14:19.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:18 smithi119 ceph-mon[40674]: osdmap e557: 8 total, 8 up, 8 in 2024-09-10T19:14:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:19 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:14:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:19 smithi119 ceph-mon[29577]: pgmap v860: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-112", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:14:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:19 smithi119 ceph-mon[29577]: osdmap e558: 8 total, 8 up, 8 in 2024-09-10T19:14:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-112"}]: dispatch 2024-09-10T19:14:20.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:19 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:14:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:19 smithi119 ceph-mon[40674]: pgmap v860: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-112", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:14:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:19 smithi119 ceph-mon[40674]: osdmap e558: 8 total, 8 up, 8 in 2024-09-10T19:14:20.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-112"}]: dispatch 2024-09-10T19:14:20.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:19 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:20.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:14:20.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:19 smithi194 ceph-mon[32464]: pgmap v860: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:20.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-112", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:14:20.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:19 smithi194 ceph-mon[32464]: osdmap e558: 8 total, 8 up, 8 in 2024-09-10T19:14:20.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-112"}]: dispatch 2024-09-10T19:14:20.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:14:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:14:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:14:22.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-112"}]': finished 2024-09-10T19:14:22.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:21 smithi119 ceph-mon[29577]: osdmap e559: 8 total, 8 up, 8 in 2024-09-10T19:14:22.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:22.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:21 smithi119 ceph-mon[29577]: pgmap v863: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:22.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-112"}]': finished 2024-09-10T19:14:22.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:21 smithi119 ceph-mon[40674]: osdmap e559: 8 total, 8 up, 8 in 2024-09-10T19:14:22.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:22.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:21 smithi119 ceph-mon[40674]: pgmap v863: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:22.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-112"}]': finished 2024-09-10T19:14:22.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:21 smithi194 ceph-mon[32464]: osdmap e559: 8 total, 8 up, 8 in 2024-09-10T19:14:22.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:22.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:21 smithi194 ceph-mon[32464]: pgmap v863: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:23.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:14:23.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:22 smithi119 ceph-mon[29577]: osdmap e560: 8 total, 8 up, 8 in 2024-09-10T19:14:23.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-112"}]: dispatch 2024-09-10T19:14:23.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:22 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:23.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:14:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:14:22] "GET /metrics HTTP/1.1" 200 226092 "" "Prometheus/2.43.0" 2024-09-10T19:14:23.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:14:23.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:22 smithi119 ceph-mon[40674]: osdmap e560: 8 total, 8 up, 8 in 2024-09-10T19:14:23.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-112"}]: dispatch 2024-09-10T19:14:23.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:22 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:23.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:14:23.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:22 smithi194 ceph-mon[32464]: osdmap e560: 8 total, 8 up, 8 in 2024-09-10T19:14:23.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-112"}]: dispatch 2024-09-10T19:14:23.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:22 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:24.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-112"}]': finished 2024-09-10T19:14:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:23 smithi119 ceph-mon[29577]: osdmap e561: 8 total, 8 up, 8 in 2024-09-10T19:14:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:23 smithi119 ceph-mon[29577]: pgmap v866: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:14:24.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-112"}]': finished 2024-09-10T19:14:24.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:23 smithi119 ceph-mon[40674]: osdmap e561: 8 total, 8 up, 8 in 2024-09-10T19:14:24.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:23 smithi119 ceph-mon[40674]: pgmap v866: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:14:24.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-112"}]': finished 2024-09-10T19:14:24.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:23 smithi194 ceph-mon[32464]: osdmap e561: 8 total, 8 up, 8 in 2024-09-10T19:14:24.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:23 smithi194 ceph-mon[32464]: pgmap v866: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:14:25.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:24 smithi194 ceph-mon[32464]: osdmap e562: 8 total, 8 up, 8 in 2024-09-10T19:14:25.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:24 smithi119 ceph-mon[29577]: osdmap e562: 8 total, 8 up, 8 in 2024-09-10T19:14:25.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:24 smithi119 ceph-mon[40674]: osdmap e562: 8 total, 8 up, 8 in 2024-09-10T19:14:26.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:25 smithi194 ceph-mon[32464]: osdmap e563: 8 total, 8 up, 8 in 2024-09-10T19:14:26.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:25 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:14:26.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:25 smithi194 ceph-mon[32464]: pgmap v869: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:14:26.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:25 smithi119 ceph-mon[29577]: osdmap e563: 8 total, 8 up, 8 in 2024-09-10T19:14:26.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:25 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:14:26.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:25 smithi119 ceph-mon[29577]: pgmap v869: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:14:26.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:25 smithi119 ceph-mon[40674]: osdmap e563: 8 total, 8 up, 8 in 2024-09-10T19:14:26.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:25 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:14:26.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:25 smithi119 ceph-mon[40674]: pgmap v869: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:14:27.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:14:27.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:26 smithi194 ceph-mon[32464]: osdmap e564: 8 total, 8 up, 8 in 2024-09-10T19:14:27.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:14:27.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:14:27.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:26 smithi119 ceph-mon[29577]: osdmap e564: 8 total, 8 up, 8 in 2024-09-10T19:14:27.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:14:27.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:14:27.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:26 smithi119 ceph-mon[40674]: osdmap e564: 8 total, 8 up, 8 in 2024-09-10T19:14:27.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:14:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-114", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:14:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:27 smithi194 ceph-mon[32464]: osdmap e565: 8 total, 8 up, 8 in 2024-09-10T19:14:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-114"}]: dispatch 2024-09-10T19:14:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:27 smithi194 ceph-mon[32464]: pgmap v872: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:27 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-114"}]': finished 2024-09-10T19:14:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:27 smithi194 ceph-mon[32464]: osdmap e566: 8 total, 8 up, 8 in 2024-09-10T19:14:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-114", "mode": "writeback"}]: dispatch 2024-09-10T19:14:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-114", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:14:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[29577]: osdmap e565: 8 total, 8 up, 8 in 2024-09-10T19:14:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-114"}]: dispatch 2024-09-10T19:14:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[29577]: pgmap v872: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-114"}]': finished 2024-09-10T19:14:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[29577]: osdmap e566: 8 total, 8 up, 8 in 2024-09-10T19:14:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-114", "mode": "writeback"}]: dispatch 2024-09-10T19:14:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-114", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:14:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[40674]: osdmap e565: 8 total, 8 up, 8 in 2024-09-10T19:14:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-114"}]: dispatch 2024-09-10T19:14:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[40674]: pgmap v872: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-114"}]': finished 2024-09-10T19:14:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[40674]: osdmap e566: 8 total, 8 up, 8 in 2024-09-10T19:14:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-114", "mode": "writeback"}]: dispatch 2024-09-10T19:14:29.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:28 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:14:29.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-114", "mode": "writeback"}]': finished 2024-09-10T19:14:29.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:28 smithi194 ceph-mon[32464]: osdmap e567: 8 total, 8 up, 8 in 2024-09-10T19:14:29.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:14:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:14:29.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:28 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:14:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-114", "mode": "writeback"}]': finished 2024-09-10T19:14:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:28 smithi119 ceph-mon[29577]: osdmap e567: 8 total, 8 up, 8 in 2024-09-10T19:14:29.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:28 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:14:29.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-114", "mode": "writeback"}]': finished 2024-09-10T19:14:29.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:28 smithi119 ceph-mon[40674]: osdmap e567: 8 total, 8 up, 8 in 2024-09-10T19:14:30.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:29 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:30.128 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:30.128 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:29 smithi194 ceph-mon[32464]: pgmap v875: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:14:30.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:29 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:30.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:30.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:29 smithi119 ceph-mon[29577]: pgmap v875: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:14:30.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:29 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:30.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:30.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:29 smithi119 ceph-mon[40674]: pgmap v875: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:14:30.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:14:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:14:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:14:31.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:30 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:14:31.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:30 smithi194 ceph-mon[32464]: osdmap e568: 8 total, 8 up, 8 in 2024-09-10T19:14:31.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:30 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-114"}]: dispatch 2024-09-10T19:14:31.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:30 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:14:31.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:30 smithi119 ceph-mon[29577]: osdmap e568: 8 total, 8 up, 8 in 2024-09-10T19:14:31.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:30 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-114"}]: dispatch 2024-09-10T19:14:31.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:30 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:14:31.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:30 smithi119 ceph-mon[40674]: osdmap e568: 8 total, 8 up, 8 in 2024-09-10T19:14:31.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:30 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-114"}]: dispatch 2024-09-10T19:14:32.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:31 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:14:32.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-114"}]': finished 2024-09-10T19:14:32.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:31 smithi194 ceph-mon[32464]: osdmap e569: 8 total, 8 up, 8 in 2024-09-10T19:14:32.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:31 smithi194 ceph-mon[32464]: pgmap v878: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:14:32.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:31 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:14:32.316 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-114"}]': finished 2024-09-10T19:14:32.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:31 smithi119 ceph-mon[40674]: osdmap e569: 8 total, 8 up, 8 in 2024-09-10T19:14:32.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:31 smithi119 ceph-mon[40674]: pgmap v878: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:14:32.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:31 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:14:32.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-114"}]': finished 2024-09-10T19:14:32.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:31 smithi119 ceph-mon[29577]: osdmap e569: 8 total, 8 up, 8 in 2024-09-10T19:14:32.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:31 smithi119 ceph-mon[29577]: pgmap v878: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:14:33.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:32 smithi119 ceph-mon[29577]: osdmap e570: 8 total, 8 up, 8 in 2024-09-10T19:14:33.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:33.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:14:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:14:32] "GET /metrics HTTP/1.1" 200 226128 "" "Prometheus/2.43.0" 2024-09-10T19:14:33.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:32 smithi119 ceph-mon[40674]: osdmap e570: 8 total, 8 up, 8 in 2024-09-10T19:14:33.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:33.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:32 smithi194 ceph-mon[32464]: osdmap e570: 8 total, 8 up, 8 in 2024-09-10T19:14:33.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:34.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:33 smithi194 ceph-mon[32464]: osdmap e571: 8 total, 8 up, 8 in 2024-09-10T19:14:34.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:14:34.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:33 smithi194 ceph-mon[32464]: pgmap v881: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:33 smithi119 ceph-mon[29577]: osdmap e571: 8 total, 8 up, 8 in 2024-09-10T19:14:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:14:34.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:33 smithi119 ceph-mon[29577]: pgmap v881: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:34.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:33 smithi119 ceph-mon[40674]: osdmap e571: 8 total, 8 up, 8 in 2024-09-10T19:14:34.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:14:34.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:33 smithi119 ceph-mon[40674]: pgmap v881: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:35.272 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:14:35.366 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:34 smithi194 ceph-mon[32464]: osdmap e572: 8 total, 8 up, 8 in 2024-09-10T19:14:35.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:14:35.366 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:34 smithi119 ceph-mon[29577]: osdmap e572: 8 total, 8 up, 8 in 2024-09-10T19:14:35.366 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:14:35.367 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:34 smithi119 ceph-mon[40674]: osdmap e572: 8 total, 8 up, 8 in 2024-09-10T19:14:36.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:35 smithi194 ceph-mon[32464]: osdmap e573: 8 total, 8 up, 8 in 2024-09-10T19:14:36.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:14:36.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:35 smithi194 ceph-mon[32464]: pgmap v884: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:36.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:35 smithi119 ceph-mon[29577]: osdmap e573: 8 total, 8 up, 8 in 2024-09-10T19:14:36.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:14:36.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:35 smithi119 ceph-mon[29577]: pgmap v884: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:36.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:35 smithi119 ceph-mon[40674]: osdmap e573: 8 total, 8 up, 8 in 2024-09-10T19:14:36.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:14:36.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:35 smithi119 ceph-mon[40674]: pgmap v884: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:14:37.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:36 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:37.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-116", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:14:37.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:36 smithi194 ceph-mon[32464]: osdmap e574: 8 total, 8 up, 8 in 2024-09-10T19:14:37.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-116"}]: dispatch 2024-09-10T19:14:37.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:36 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:37.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-116", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:14:37.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:36 smithi119 ceph-mon[29577]: osdmap e574: 8 total, 8 up, 8 in 2024-09-10T19:14:37.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-116"}]: dispatch 2024-09-10T19:14:37.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:36 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:14:37.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-116", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:14:37.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:36 smithi119 ceph-mon[40674]: osdmap e574: 8 total, 8 up, 8 in 2024-09-10T19:14:37.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-116"}]: dispatch 2024-09-10T19:14:38.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-116"}]': finished 2024-09-10T19:14:38.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:37 smithi194 ceph-mon[32464]: osdmap e575: 8 total, 8 up, 8 in 2024-09-10T19:14:38.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-116", "mode": "writeback"}]: dispatch 2024-09-10T19:14:38.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:37 smithi194 ceph-mon[32464]: pgmap v887: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:38.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-116"}]': finished 2024-09-10T19:14:38.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:37 smithi119 ceph-mon[29577]: osdmap e575: 8 total, 8 up, 8 in 2024-09-10T19:14:38.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-116", "mode": "writeback"}]: dispatch 2024-09-10T19:14:38.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:37 smithi119 ceph-mon[29577]: pgmap v887: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-116"}]': finished 2024-09-10T19:14:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:37 smithi119 ceph-mon[40674]: osdmap e575: 8 total, 8 up, 8 in 2024-09-10T19:14:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-116", "mode": "writeback"}]: dispatch 2024-09-10T19:14:38.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:37 smithi119 ceph-mon[40674]: pgmap v887: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:39.019 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:14:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:14:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:38 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:14:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-116", "mode": "writeback"}]': finished 2024-09-10T19:14:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:38 smithi194 ceph-mon[32464]: osdmap e576: 8 total, 8 up, 8 in 2024-09-10T19:14:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "244.4"}]: dispatch 2024-09-10T19:14:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:38 smithi194 ceph-mon[32464]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "244.4"}]: dispatch 2024-09-10T19:14:39.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:38 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:14:39.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-116", "mode": "writeback"}]': finished 2024-09-10T19:14:39.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:38 smithi119 ceph-mon[29577]: osdmap e576: 8 total, 8 up, 8 in 2024-09-10T19:14:39.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "244.4"}]: dispatch 2024-09-10T19:14:39.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:38 smithi119 ceph-mon[29577]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "244.4"}]: dispatch 2024-09-10T19:14:39.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:38 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:14:39.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-116", "mode": "writeback"}]': finished 2024-09-10T19:14:39.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:38 smithi119 ceph-mon[40674]: osdmap e576: 8 total, 8 up, 8 in 2024-09-10T19:14:39.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "244.4"}]: dispatch 2024-09-10T19:14:39.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:38 smithi119 ceph-mon[40674]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "244.4"}]: dispatch 2024-09-10T19:14:40.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:39 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:40.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:39 smithi194 ceph-mon[32464]: pgmap v889: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-10T19:14:40.269 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:14:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:14:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:14:40.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:39 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:40.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:39 smithi119 ceph-mon[29577]: pgmap v889: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-10T19:14:40.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:39 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:40.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:39 smithi119 ceph-mon[40674]: pgmap v889: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-09-10T19:14:42.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:42 smithi119 ceph-mon[29577]: pgmap v890: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:14:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:42 smithi119 ceph-mon[29577]: 244.4 scrub starts 2024-09-10T19:14:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:42 smithi119 ceph-mon[29577]: 244.4 scrub ok 2024-09-10T19:14:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:42 smithi119 ceph-mon[40674]: pgmap v890: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:14:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:42 smithi119 ceph-mon[40674]: 244.4 scrub starts 2024-09-10T19:14:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:42 smithi119 ceph-mon[40674]: 244.4 scrub ok 2024-09-10T19:14:42.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:42 smithi194 ceph-mon[32464]: pgmap v890: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:14:42.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:42 smithi194 ceph-mon[32464]: 244.4 scrub starts 2024-09-10T19:14:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:42 smithi194 ceph-mon[32464]: 244.4 scrub ok 2024-09-10T19:14:43.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:14:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:14:42] "GET /metrics HTTP/1.1" 200 226128 "" "Prometheus/2.43.0" 2024-09-10T19:14:44.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:44 smithi119 ceph-mon[29577]: pgmap v891: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 974 B/s wr, 2 op/s 2024-09-10T19:14:44.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:44 smithi119 ceph-mon[40674]: pgmap v891: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 974 B/s wr, 2 op/s 2024-09-10T19:14:44.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:44 smithi194 ceph-mon[32464]: pgmap v891: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 974 B/s wr, 2 op/s 2024-09-10T19:14:46.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:46 smithi194 ceph-mon[32464]: pgmap v892: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 858 B/s wr, 2 op/s 2024-09-10T19:14:46.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:46 smithi119 ceph-mon[29577]: pgmap v892: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 858 B/s wr, 2 op/s 2024-09-10T19:14:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:46 smithi119 ceph-mon[40674]: pgmap v892: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 858 B/s wr, 2 op/s 2024-09-10T19:14:48.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:48 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:14:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:48 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:48 smithi194 ceph-mon[32464]: pgmap v893: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 716 B/s wr, 2 op/s 2024-09-10T19:14:48.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:48 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:14:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:48 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:48 smithi119 ceph-mon[29577]: pgmap v893: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 716 B/s wr, 2 op/s 2024-09-10T19:14:48.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:48 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:14:48.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:48 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:48.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:48 smithi119 ceph-mon[40674]: pgmap v893: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 716 B/s wr, 2 op/s 2024-09-10T19:14:49.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:14:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:14:50.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:14:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:14:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:14:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:50 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:50 smithi194 ceph-mon[32464]: pgmap v894: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 631 B/s wr, 2 op/s 2024-09-10T19:14:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:50 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:50.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:50 smithi119 ceph-mon[29577]: pgmap v894: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 631 B/s wr, 2 op/s 2024-09-10T19:14:50.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:50 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:50.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:50 smithi119 ceph-mon[40674]: pgmap v894: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 631 B/s wr, 2 op/s 2024-09-10T19:14:51.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:51 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:14:51.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:51 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:14:51.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:51 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:14:51.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:51 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:14:51.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:51 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:14:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:51 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:14:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:51 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:14:51.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:51 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:14:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:51 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:14:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:51 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:14:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:51 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:14:51.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:51 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:14:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:52 smithi119 ceph-mon[29577]: pgmap v895: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:14:52.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:52 smithi119 ceph-mon[40674]: pgmap v895: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:14:52.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:52 smithi194 ceph-mon[32464]: pgmap v895: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:14:53.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:14:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:14:52] "GET /metrics HTTP/1.1" 200 228599 "" "Prometheus/2.43.0" 2024-09-10T19:14:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:54 smithi194 ceph-mon[32464]: pgmap v896: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:14:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:54 smithi194 ceph-mon[32464]: osdmap e577: 8 total, 8 up, 8 in 2024-09-10T19:14:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:54 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:54.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:54 smithi119 ceph-mon[29577]: pgmap v896: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:14:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:54 smithi119 ceph-mon[29577]: osdmap e577: 8 total, 8 up, 8 in 2024-09-10T19:14:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:54 smithi119 ceph-mon[40674]: pgmap v896: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:14:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:54 smithi119 ceph-mon[40674]: osdmap e577: 8 total, 8 up, 8 in 2024-09-10T19:14:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:54 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:14:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:14:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:55 smithi194 ceph-mon[32464]: osdmap e578: 8 total, 8 up, 8 in 2024-09-10T19:14:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-116"}]: dispatch 2024-09-10T19:14:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:55 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:14:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-116"}]': finished 2024-09-10T19:14:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:55 smithi194 ceph-mon[32464]: osdmap e579: 8 total, 8 up, 8 in 2024-09-10T19:14:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:14:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:55 smithi119 ceph-mon[29577]: osdmap e578: 8 total, 8 up, 8 in 2024-09-10T19:14:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-116"}]: dispatch 2024-09-10T19:14:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:55 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:14:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-116"}]': finished 2024-09-10T19:14:55.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:55 smithi119 ceph-mon[29577]: osdmap e579: 8 total, 8 up, 8 in 2024-09-10T19:14:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:14:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:55 smithi119 ceph-mon[40674]: osdmap e578: 8 total, 8 up, 8 in 2024-09-10T19:14:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-116"}]: dispatch 2024-09-10T19:14:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:55 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:14:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-116"}]': finished 2024-09-10T19:14:55.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:55 smithi119 ceph-mon[40674]: osdmap e579: 8 total, 8 up, 8 in 2024-09-10T19:14:57.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:56 smithi194 ceph-mon[32464]: pgmap v899: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-09-10T19:14:57.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:56 smithi119 ceph-mon[29577]: pgmap v899: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-09-10T19:14:57.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:56 smithi119 ceph-mon[40674]: pgmap v899: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 2 op/s 2024-09-10T19:14:58.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:57 smithi194 ceph-mon[32464]: osdmap e580: 8 total, 8 up, 8 in 2024-09-10T19:14:58.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:57 smithi194 ceph-mon[32464]: osdmap e581: 8 total, 8 up, 8 in 2024-09-10T19:14:58.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:14:58.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:57 smithi119 ceph-mon[29577]: osdmap e580: 8 total, 8 up, 8 in 2024-09-10T19:14:58.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:57 smithi119 ceph-mon[29577]: osdmap e581: 8 total, 8 up, 8 in 2024-09-10T19:14:58.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:14:58.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:57 smithi119 ceph-mon[40674]: osdmap e580: 8 total, 8 up, 8 in 2024-09-10T19:14:58.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:57 smithi119 ceph-mon[40674]: osdmap e581: 8 total, 8 up, 8 in 2024-09-10T19:14:58.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:14:59.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:14:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:14:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:58 smithi194 ceph-mon[32464]: pgmap v902: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:14:59.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:14:58 smithi194 ceph-mon[32464]: osdmap e582: 8 total, 8 up, 8 in 2024-09-10T19:14:59.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:58 smithi119 ceph-mon[29577]: pgmap v902: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:59.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:14:59.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:14:58 smithi119 ceph-mon[29577]: osdmap e582: 8 total, 8 up, 8 in 2024-09-10T19:14:59.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:58 smithi119 ceph-mon[40674]: pgmap v902: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:14:59.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:14:59.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:14:58 smithi119 ceph-mon[40674]: osdmap e582: 8 total, 8 up, 8 in 2024-09-10T19:15:00.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:15:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:15:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:15:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:00 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:00 smithi194 ceph-mon[32464]: pgmap v905: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:00 smithi194 ceph-mon[32464]: osdmap e583: 8 total, 8 up, 8 in 2024-09-10T19:15:00.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:00 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:00 smithi119 ceph-mon[29577]: pgmap v905: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:00 smithi119 ceph-mon[29577]: osdmap e583: 8 total, 8 up, 8 in 2024-09-10T19:15:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:00 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:00 smithi119 ceph-mon[40674]: pgmap v905: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:00 smithi119 ceph-mon[40674]: osdmap e583: 8 total, 8 up, 8 in 2024-09-10T19:15:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:01.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-118", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:01.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:01 smithi194 ceph-mon[32464]: osdmap e584: 8 total, 8 up, 8 in 2024-09-10T19:15:01.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-118"}]: dispatch 2024-09-10T19:15:01.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-118"}]': finished 2024-09-10T19:15:01.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:01 smithi194 ceph-mon[32464]: osdmap e585: 8 total, 8 up, 8 in 2024-09-10T19:15:01.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-118", "mode": "writeback"}]: dispatch 2024-09-10T19:15:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-118", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:01 smithi119 ceph-mon[29577]: osdmap e584: 8 total, 8 up, 8 in 2024-09-10T19:15:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-118"}]: dispatch 2024-09-10T19:15:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-118"}]': finished 2024-09-10T19:15:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:01 smithi119 ceph-mon[29577]: osdmap e585: 8 total, 8 up, 8 in 2024-09-10T19:15:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-118", "mode": "writeback"}]: dispatch 2024-09-10T19:15:01.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-118", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:01.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:01 smithi119 ceph-mon[40674]: osdmap e584: 8 total, 8 up, 8 in 2024-09-10T19:15:01.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-118"}]: dispatch 2024-09-10T19:15:01.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-118"}]': finished 2024-09-10T19:15:01.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:01 smithi119 ceph-mon[40674]: osdmap e585: 8 total, 8 up, 8 in 2024-09-10T19:15:01.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-118", "mode": "writeback"}]: dispatch 2024-09-10T19:15:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:02 smithi119 ceph-mon[29577]: pgmap v908: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:15:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:15:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:02 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:02 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-118", "mode": "writeback"}]': finished 2024-09-10T19:15:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:02 smithi119 ceph-mon[29577]: osdmap e586: 8 total, 8 up, 8 in 2024-09-10T19:15:02.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:02 smithi119 ceph-mon[40674]: pgmap v908: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:15:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:15:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:02 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:02 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-118", "mode": "writeback"}]': finished 2024-09-10T19:15:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:02 smithi119 ceph-mon[40674]: osdmap e586: 8 total, 8 up, 8 in 2024-09-10T19:15:02.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:02 smithi194 ceph-mon[32464]: pgmap v908: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:15:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:15:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:02 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:02 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-118", "mode": "writeback"}]': finished 2024-09-10T19:15:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:02 smithi194 ceph-mon[32464]: osdmap e586: 8 total, 8 up, 8 in 2024-09-10T19:15:03.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:15:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:15:02] "GET /metrics HTTP/1.1" 200 228555 "" "Prometheus/2.43.0" 2024-09-10T19:15:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:04 smithi119 ceph-mon[29577]: pgmap v911: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:15:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:04 smithi119 ceph-mon[29577]: osdmap e587: 8 total, 8 up, 8 in 2024-09-10T19:15:04.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:04.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:04 smithi119 ceph-mon[40674]: pgmap v911: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:15:04.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:04 smithi119 ceph-mon[40674]: osdmap e587: 8 total, 8 up, 8 in 2024-09-10T19:15:04.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:04.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:04 smithi194 ceph-mon[32464]: pgmap v911: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:15:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:04 smithi194 ceph-mon[32464]: osdmap e587: 8 total, 8 up, 8 in 2024-09-10T19:15:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:05 smithi119 ceph-mon[29577]: osdmap e588: 8 total, 8 up, 8 in 2024-09-10T19:15:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-118"}]: dispatch 2024-09-10T19:15:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:05 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:05.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:05 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-118"}]': finished 2024-09-10T19:15:05.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:05.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:05 smithi119 ceph-mon[40674]: osdmap e588: 8 total, 8 up, 8 in 2024-09-10T19:15:05.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-118"}]: dispatch 2024-09-10T19:15:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:05 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:05.568 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:05 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-118"}]': finished 2024-09-10T19:15:05.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:05.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:05 smithi194 ceph-mon[32464]: osdmap e588: 8 total, 8 up, 8 in 2024-09-10T19:15:05.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-118"}]: dispatch 2024-09-10T19:15:05.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:05 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:05.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:05 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-118"}]': finished 2024-09-10T19:15:06.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:06 smithi119 ceph-mon[29577]: pgmap v914: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:15:06.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:06 smithi119 ceph-mon[29577]: osdmap e589: 8 total, 8 up, 8 in 2024-09-10T19:15:06.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:06 smithi119 ceph-mon[40674]: pgmap v914: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:15:06.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:06 smithi119 ceph-mon[40674]: osdmap e589: 8 total, 8 up, 8 in 2024-09-10T19:15:06.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:06 smithi194 ceph-mon[32464]: pgmap v914: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:15:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:06 smithi194 ceph-mon[32464]: osdmap e589: 8 total, 8 up, 8 in 2024-09-10T19:15:07.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:07 smithi194 ceph-mon[32464]: osdmap e590: 8 total, 8 up, 8 in 2024-09-10T19:15:07.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:07 smithi119 ceph-mon[29577]: osdmap e590: 8 total, 8 up, 8 in 2024-09-10T19:15:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:07 smithi119 ceph-mon[40674]: osdmap e590: 8 total, 8 up, 8 in 2024-09-10T19:15:08.738 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:08 smithi194 ceph-mon[32464]: pgmap v917: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:08.738 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:08 smithi194 ceph-mon[32464]: osdmap e591: 8 total, 8 up, 8 in 2024-09-10T19:15:08.738 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:08 smithi119 ceph-mon[29577]: pgmap v917: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:08 smithi119 ceph-mon[29577]: osdmap e591: 8 total, 8 up, 8 in 2024-09-10T19:15:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:08 smithi119 ceph-mon[40674]: pgmap v917: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:08 smithi119 ceph-mon[40674]: osdmap e591: 8 total, 8 up, 8 in 2024-09-10T19:15:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:09.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:15:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:15:09.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:09.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:09 smithi194 ceph-mon[32464]: osdmap e592: 8 total, 8 up, 8 in 2024-09-10T19:15:09.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:09 smithi119 ceph-mon[29577]: osdmap e592: 8 total, 8 up, 8 in 2024-09-10T19:15:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:09 smithi119 ceph-mon[40674]: osdmap e592: 8 total, 8 up, 8 in 2024-09-10T19:15:09.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:10.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:15:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:15:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:15:10.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:10 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:10.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:10 smithi194 ceph-mon[32464]: pgmap v920: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:15:10.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-120", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:10.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:10 smithi194 ceph-mon[32464]: osdmap e593: 8 total, 8 up, 8 in 2024-09-10T19:15:10.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-120"}]: dispatch 2024-09-10T19:15:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:10 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:10 smithi119 ceph-mon[29577]: pgmap v920: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:15:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-120", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:10 smithi119 ceph-mon[29577]: osdmap e593: 8 total, 8 up, 8 in 2024-09-10T19:15:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-120"}]: dispatch 2024-09-10T19:15:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:10 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:10 smithi119 ceph-mon[40674]: pgmap v920: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:15:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-120", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:10 smithi119 ceph-mon[40674]: osdmap e593: 8 total, 8 up, 8 in 2024-09-10T19:15:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-120"}]: dispatch 2024-09-10T19:15:11.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-120"}]': finished 2024-09-10T19:15:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:11 smithi194 ceph-mon[32464]: osdmap e594: 8 total, 8 up, 8 in 2024-09-10T19:15:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-120", "mode": "writeback"}]: dispatch 2024-09-10T19:15:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:11 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-120"}]': finished 2024-09-10T19:15:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:11 smithi119 ceph-mon[29577]: osdmap e594: 8 total, 8 up, 8 in 2024-09-10T19:15:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-120", "mode": "writeback"}]: dispatch 2024-09-10T19:15:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:11 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-120"}]': finished 2024-09-10T19:15:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:11 smithi119 ceph-mon[40674]: osdmap e594: 8 total, 8 up, 8 in 2024-09-10T19:15:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-120", "mode": "writeback"}]: dispatch 2024-09-10T19:15:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:11 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:12.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-mon[29577]: pgmap v923: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:15:12.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-120", "mode": "writeback"}]': finished 2024-09-10T19:15:12.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-mon[29577]: osdmap e595: 8 total, 8 up, 8 in 2024-09-10T19:15:12.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:12.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:12.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-mon[29577]: osdmap e596: 8 total, 8 up, 8 in 2024-09-10T19:15:12.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-120"}]: dispatch 2024-09-10T19:15:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-mon[40674]: pgmap v923: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:15:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-120", "mode": "writeback"}]': finished 2024-09-10T19:15:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-mon[40674]: osdmap e595: 8 total, 8 up, 8 in 2024-09-10T19:15:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:12.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-mon[40674]: osdmap e596: 8 total, 8 up, 8 in 2024-09-10T19:15:12.706 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-120"}]: dispatch 2024-09-10T19:15:12.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:12 smithi194 ceph-mon[32464]: pgmap v923: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:15:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-120", "mode": "writeback"}]': finished 2024-09-10T19:15:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:12 smithi194 ceph-mon[32464]: osdmap e595: 8 total, 8 up, 8 in 2024-09-10T19:15:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:12 smithi194 ceph-mon[32464]: osdmap e596: 8 total, 8 up, 8 in 2024-09-10T19:15:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-120"}]: dispatch 2024-09-10T19:15:13.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:15:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:15:12] "GET /metrics HTTP/1.1" 200 228555 "" "Prometheus/2.43.0" 2024-09-10T19:15:13.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:13 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-120"}]': finished 2024-09-10T19:15:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:13 smithi194 ceph-mon[32464]: osdmap e597: 8 total, 8 up, 8 in 2024-09-10T19:15:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:13 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-120"}]': finished 2024-09-10T19:15:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:13 smithi119 ceph-mon[29577]: osdmap e597: 8 total, 8 up, 8 in 2024-09-10T19:15:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:13 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-120"}]': finished 2024-09-10T19:15:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:13 smithi119 ceph-mon[40674]: osdmap e597: 8 total, 8 up, 8 in 2024-09-10T19:15:14.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:14 smithi119 ceph-mon[29577]: pgmap v926: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-10T19:15:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:14 smithi119 ceph-mon[29577]: osdmap e598: 8 total, 8 up, 8 in 2024-09-10T19:15:14.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:14 smithi119 ceph-mon[40674]: pgmap v926: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-10T19:15:14.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:14 smithi119 ceph-mon[40674]: osdmap e598: 8 total, 8 up, 8 in 2024-09-10T19:15:15.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:14 smithi194 ceph-mon[32464]: pgmap v926: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-09-10T19:15:15.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:14 smithi194 ceph-mon[32464]: osdmap e598: 8 total, 8 up, 8 in 2024-09-10T19:15:16.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:16 smithi119 ceph-mon[40674]: pgmap v929: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:15:16.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:16 smithi119 ceph-mon[40674]: osdmap e599: 8 total, 8 up, 8 in 2024-09-10T19:15:16.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:16 smithi119 ceph-mon[29577]: pgmap v929: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:15:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:16 smithi119 ceph-mon[29577]: osdmap e599: 8 total, 8 up, 8 in 2024-09-10T19:15:16.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:16.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:16 smithi194 ceph-mon[32464]: pgmap v929: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:15:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:16 smithi194 ceph-mon[32464]: osdmap e599: 8 total, 8 up, 8 in 2024-09-10T19:15:16.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:17.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:17 smithi194 ceph-mon[32464]: osdmap e600: 8 total, 8 up, 8 in 2024-09-10T19:15:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-122", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:17 smithi194 ceph-mon[32464]: osdmap e601: 8 total, 8 up, 8 in 2024-09-10T19:15:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-122"}]: dispatch 2024-09-10T19:15:17.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:17 smithi119 ceph-mon[29577]: osdmap e600: 8 total, 8 up, 8 in 2024-09-10T19:15:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-122", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:17 smithi119 ceph-mon[29577]: osdmap e601: 8 total, 8 up, 8 in 2024-09-10T19:15:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-122"}]: dispatch 2024-09-10T19:15:17.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:17.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:17 smithi119 ceph-mon[40674]: osdmap e600: 8 total, 8 up, 8 in 2024-09-10T19:15:17.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:17.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-122", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:17 smithi119 ceph-mon[40674]: osdmap e601: 8 total, 8 up, 8 in 2024-09-10T19:15:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-122"}]: dispatch 2024-09-10T19:15:18.749 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:18 smithi194 ceph-mon[32464]: pgmap v932: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:18.750 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-122"}]': finished 2024-09-10T19:15:18.750 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:18 smithi194 ceph-mon[32464]: osdmap e602: 8 total, 8 up, 8 in 2024-09-10T19:15:18.750 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-122", "mode": "writeback"}]: dispatch 2024-09-10T19:15:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:18 smithi119 ceph-mon[29577]: pgmap v932: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-122"}]': finished 2024-09-10T19:15:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:18 smithi119 ceph-mon[29577]: osdmap e602: 8 total, 8 up, 8 in 2024-09-10T19:15:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-122", "mode": "writeback"}]: dispatch 2024-09-10T19:15:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:18 smithi119 ceph-mon[40674]: pgmap v932: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-122"}]': finished 2024-09-10T19:15:18.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:18 smithi119 ceph-mon[40674]: osdmap e602: 8 total, 8 up, 8 in 2024-09-10T19:15:18.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-122", "mode": "writeback"}]: dispatch 2024-09-10T19:15:19.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:15:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:15:19.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:19 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:19.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-122", "mode": "writeback"}]': finished 2024-09-10T19:15:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:19 smithi194 ceph-mon[32464]: osdmap e603: 8 total, 8 up, 8 in 2024-09-10T19:15:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:19 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-122", "mode": "writeback"}]': finished 2024-09-10T19:15:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:19 smithi119 ceph-mon[29577]: osdmap e603: 8 total, 8 up, 8 in 2024-09-10T19:15:19.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:19 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:19.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-122", "mode": "writeback"}]': finished 2024-09-10T19:15:19.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:19 smithi119 ceph-mon[40674]: osdmap e603: 8 total, 8 up, 8 in 2024-09-10T19:15:20.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:20 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:20.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:20 smithi194 ceph-mon[32464]: pgmap v935: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:20.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:20.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:15:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:15:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:15:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:20 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:20 smithi119 ceph-mon[29577]: pgmap v935: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:20 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:20 smithi119 ceph-mon[40674]: pgmap v935: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:21.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:21.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:21 smithi119 ceph-mon[29577]: osdmap e604: 8 total, 8 up, 8 in 2024-09-10T19:15:21.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:21 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-122"}]: dispatch 2024-09-10T19:15:21.718 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:21.718 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:21 smithi119 ceph-mon[40674]: osdmap e604: 8 total, 8 up, 8 in 2024-09-10T19:15:21.719 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:21 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-122"}]: dispatch 2024-09-10T19:15:21.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:21 smithi194 ceph-mon[32464]: osdmap e604: 8 total, 8 up, 8 in 2024-09-10T19:15:21.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:21 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-122"}]: dispatch 2024-09-10T19:15:23.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:22 smithi194 ceph-mon[32464]: pgmap v938: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:23.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:22 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:23.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-122"}]': finished 2024-09-10T19:15:23.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:22 smithi194 ceph-mon[32464]: osdmap e605: 8 total, 8 up, 8 in 2024-09-10T19:15:23.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:15:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:15:22] "GET /metrics HTTP/1.1" 200 228613 "" "Prometheus/2.43.0" 2024-09-10T19:15:23.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:22 smithi119 ceph-mon[29577]: pgmap v938: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:23.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:22 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:23.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-122"}]': finished 2024-09-10T19:15:23.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:22 smithi119 ceph-mon[29577]: osdmap e605: 8 total, 8 up, 8 in 2024-09-10T19:15:23.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:22 smithi119 ceph-mon[40674]: pgmap v938: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:23.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:22 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:23.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-122"}]': finished 2024-09-10T19:15:23.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:22 smithi119 ceph-mon[40674]: osdmap e605: 8 total, 8 up, 8 in 2024-09-10T19:15:24.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:23 smithi194 ceph-mon[32464]: osdmap e606: 8 total, 8 up, 8 in 2024-09-10T19:15:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:23 smithi194 ceph-mon[32464]: osdmap e607: 8 total, 8 up, 8 in 2024-09-10T19:15:24.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:24.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:23 smithi119 ceph-mon[29577]: osdmap e606: 8 total, 8 up, 8 in 2024-09-10T19:15:24.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:23 smithi119 ceph-mon[29577]: osdmap e607: 8 total, 8 up, 8 in 2024-09-10T19:15:24.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:24.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:23 smithi119 ceph-mon[40674]: osdmap e606: 8 total, 8 up, 8 in 2024-09-10T19:15:24.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:23 smithi119 ceph-mon[40674]: osdmap e607: 8 total, 8 up, 8 in 2024-09-10T19:15:24.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:25.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:24 smithi194 ceph-mon[32464]: pgmap v941: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:25.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:25.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:24 smithi194 ceph-mon[32464]: osdmap e608: 8 total, 8 up, 8 in 2024-09-10T19:15:25.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:24 smithi119 ceph-mon[29577]: pgmap v941: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:24 smithi119 ceph-mon[29577]: osdmap e608: 8 total, 8 up, 8 in 2024-09-10T19:15:25.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:24 smithi119 ceph-mon[40674]: pgmap v941: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:25.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:25.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:24 smithi119 ceph-mon[40674]: osdmap e608: 8 total, 8 up, 8 in 2024-09-10T19:15:26.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:26 smithi194 ceph-mon[32464]: pgmap v944: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:26 smithi194 ceph-mon[32464]: osdmap e609: 8 total, 8 up, 8 in 2024-09-10T19:15:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:26 smithi119 ceph-mon[40674]: pgmap v944: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:26 smithi119 ceph-mon[40674]: osdmap e609: 8 total, 8 up, 8 in 2024-09-10T19:15:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:26 smithi119 ceph-mon[29577]: pgmap v944: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:26 smithi119 ceph-mon[29577]: osdmap e609: 8 total, 8 up, 8 in 2024-09-10T19:15:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:27.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-124", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:27 smithi194 ceph-mon[32464]: osdmap e610: 8 total, 8 up, 8 in 2024-09-10T19:15:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-124"}]: dispatch 2024-09-10T19:15:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-124"}]': finished 2024-09-10T19:15:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:27 smithi194 ceph-mon[32464]: osdmap e611: 8 total, 8 up, 8 in 2024-09-10T19:15:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-124", "mode": "writeback"}]: dispatch 2024-09-10T19:15:27.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-124", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:27 smithi119 ceph-mon[29577]: osdmap e610: 8 total, 8 up, 8 in 2024-09-10T19:15:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-124"}]: dispatch 2024-09-10T19:15:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-124"}]': finished 2024-09-10T19:15:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:27 smithi119 ceph-mon[29577]: osdmap e611: 8 total, 8 up, 8 in 2024-09-10T19:15:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-124", "mode": "writeback"}]: dispatch 2024-09-10T19:15:27.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-124", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:27.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:27 smithi119 ceph-mon[40674]: osdmap e610: 8 total, 8 up, 8 in 2024-09-10T19:15:27.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-124"}]: dispatch 2024-09-10T19:15:27.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-124"}]': finished 2024-09-10T19:15:27.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:27 smithi119 ceph-mon[40674]: osdmap e611: 8 total, 8 up, 8 in 2024-09-10T19:15:27.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-124", "mode": "writeback"}]: dispatch 2024-09-10T19:15:28.760 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:28 smithi194 ceph-mon[32464]: pgmap v947: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:28.761 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:28 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:28.761 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-124", "mode": "writeback"}]': finished 2024-09-10T19:15:28.761 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:28 smithi194 ceph-mon[32464]: osdmap e612: 8 total, 8 up, 8 in 2024-09-10T19:15:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:28 smithi119 ceph-mon[40674]: pgmap v947: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:28 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-124", "mode": "writeback"}]': finished 2024-09-10T19:15:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:28 smithi119 ceph-mon[40674]: osdmap e612: 8 total, 8 up, 8 in 2024-09-10T19:15:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:28 smithi119 ceph-mon[29577]: pgmap v947: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:28 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-124", "mode": "writeback"}]': finished 2024-09-10T19:15:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:28 smithi119 ceph-mon[29577]: osdmap e612: 8 total, 8 up, 8 in 2024-09-10T19:15:29.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:15:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:15:30.390 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:15:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:15:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:15:30.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:30 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:30.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:30 smithi194 ceph-mon[32464]: pgmap v950: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:15:30.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:30 smithi194 ceph-mon[32464]: osdmap e613: 8 total, 8 up, 8 in 2024-09-10T19:15:30.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:30 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:30.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:30 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:30 smithi119 ceph-mon[29577]: pgmap v950: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:15:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:30 smithi119 ceph-mon[29577]: osdmap e613: 8 total, 8 up, 8 in 2024-09-10T19:15:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:30 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:30 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:30 smithi119 ceph-mon[40674]: pgmap v950: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:15:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:30 smithi119 ceph-mon[40674]: osdmap e613: 8 total, 8 up, 8 in 2024-09-10T19:15:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:30 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:31 smithi194 ceph-mon[32464]: osdmap e614: 8 total, 8 up, 8 in 2024-09-10T19:15:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-124"}]: dispatch 2024-09-10T19:15:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:31 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:31 smithi119 ceph-mon[29577]: osdmap e614: 8 total, 8 up, 8 in 2024-09-10T19:15:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-124"}]: dispatch 2024-09-10T19:15:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:31 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:31 smithi119 ceph-mon[40674]: osdmap e614: 8 total, 8 up, 8 in 2024-09-10T19:15:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-124"}]: dispatch 2024-09-10T19:15:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:31 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:32.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:32 smithi119 ceph-mon[40674]: pgmap v953: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:15:32.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-124"}]': finished 2024-09-10T19:15:32.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:32 smithi119 ceph-mon[40674]: osdmap e615: 8 total, 8 up, 8 in 2024-09-10T19:15:32.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:32.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:32 smithi119 ceph-mon[29577]: pgmap v953: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:15:32.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-124"}]': finished 2024-09-10T19:15:32.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:32 smithi119 ceph-mon[29577]: osdmap e615: 8 total, 8 up, 8 in 2024-09-10T19:15:32.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:32 smithi194 ceph-mon[32464]: pgmap v953: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-09-10T19:15:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-124"}]': finished 2024-09-10T19:15:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:32 smithi194 ceph-mon[32464]: osdmap e615: 8 total, 8 up, 8 in 2024-09-10T19:15:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:33.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:15:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:15:32] "GET /metrics HTTP/1.1" 200 228593 "" "Prometheus/2.43.0" 2024-09-10T19:15:33.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:33 smithi194 ceph-mon[32464]: osdmap e616: 8 total, 8 up, 8 in 2024-09-10T19:15:33.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:33 smithi119 ceph-mon[29577]: osdmap e616: 8 total, 8 up, 8 in 2024-09-10T19:15:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:33 smithi119 ceph-mon[40674]: osdmap e616: 8 total, 8 up, 8 in 2024-09-10T19:15:34.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:34 smithi194 ceph-mon[32464]: pgmap v956: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-10T19:15:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:34 smithi194 ceph-mon[32464]: osdmap e617: 8 total, 8 up, 8 in 2024-09-10T19:15:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:34 smithi119 ceph-mon[29577]: pgmap v956: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-10T19:15:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:34 smithi119 ceph-mon[29577]: osdmap e617: 8 total, 8 up, 8 in 2024-09-10T19:15:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:34 smithi119 ceph-mon[40674]: pgmap v956: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-10T19:15:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:34 smithi119 ceph-mon[40674]: osdmap e617: 8 total, 8 up, 8 in 2024-09-10T19:15:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:35 smithi194 ceph-mon[32464]: osdmap e618: 8 total, 8 up, 8 in 2024-09-10T19:15:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:35.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:35 smithi119 ceph-mon[29577]: osdmap e618: 8 total, 8 up, 8 in 2024-09-10T19:15:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:35 smithi119 ceph-mon[40674]: osdmap e618: 8 total, 8 up, 8 in 2024-09-10T19:15:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:36.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:36 smithi194 ceph-mon[32464]: pgmap v959: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-10T19:15:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-126", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:36 smithi194 ceph-mon[32464]: osdmap e619: 8 total, 8 up, 8 in 2024-09-10T19:15:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-126"}]: dispatch 2024-09-10T19:15:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-126"}]': finished 2024-09-10T19:15:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:36 smithi194 ceph-mon[32464]: osdmap e620: 8 total, 8 up, 8 in 2024-09-10T19:15:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-126", "mode": "writeback"}]: dispatch 2024-09-10T19:15:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:36 smithi119 ceph-mon[29577]: pgmap v959: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-10T19:15:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-126", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:36 smithi119 ceph-mon[29577]: osdmap e619: 8 total, 8 up, 8 in 2024-09-10T19:15:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-126"}]: dispatch 2024-09-10T19:15:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-126"}]': finished 2024-09-10T19:15:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:36 smithi119 ceph-mon[29577]: osdmap e620: 8 total, 8 up, 8 in 2024-09-10T19:15:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-126", "mode": "writeback"}]: dispatch 2024-09-10T19:15:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:36 smithi119 ceph-mon[40674]: pgmap v959: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 1 op/s 2024-09-10T19:15:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-126", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:36 smithi119 ceph-mon[40674]: osdmap e619: 8 total, 8 up, 8 in 2024-09-10T19:15:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-126"}]: dispatch 2024-09-10T19:15:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-126"}]': finished 2024-09-10T19:15:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:36 smithi119 ceph-mon[40674]: osdmap e620: 8 total, 8 up, 8 in 2024-09-10T19:15:36.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-126", "mode": "writeback"}]: dispatch 2024-09-10T19:15:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:37 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-126", "mode": "writeback"}]': finished 2024-09-10T19:15:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:37 smithi194 ceph-mon[32464]: osdmap e621: 8 total, 8 up, 8 in 2024-09-10T19:15:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:37.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:37 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-126", "mode": "writeback"}]': finished 2024-09-10T19:15:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:37 smithi119 ceph-mon[29577]: osdmap e621: 8 total, 8 up, 8 in 2024-09-10T19:15:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:37.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:37 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:37.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-126", "mode": "writeback"}]': finished 2024-09-10T19:15:37.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:37 smithi119 ceph-mon[40674]: osdmap e621: 8 total, 8 up, 8 in 2024-09-10T19:15:37.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:38.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:38 smithi194 ceph-mon[32464]: pgmap v962: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:38 smithi194 ceph-mon[32464]: osdmap e622: 8 total, 8 up, 8 in 2024-09-10T19:15:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-126"}]: dispatch 2024-09-10T19:15:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:38 smithi119 ceph-mon[29577]: pgmap v962: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:38 smithi119 ceph-mon[29577]: osdmap e622: 8 total, 8 up, 8 in 2024-09-10T19:15:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-126"}]: dispatch 2024-09-10T19:15:38.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:38 smithi119 ceph-mon[40674]: pgmap v962: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:38.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:38 smithi119 ceph-mon[40674]: osdmap e622: 8 total, 8 up, 8 in 2024-09-10T19:15:38.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-126"}]: dispatch 2024-09-10T19:15:39.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:15:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:15:39.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:39 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:39.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:39 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-126"}]': finished 2024-09-10T19:15:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:39 smithi194 ceph-mon[32464]: osdmap e623: 8 total, 8 up, 8 in 2024-09-10T19:15:39.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:39 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:39.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:39 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-126"}]': finished 2024-09-10T19:15:39.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:39 smithi119 ceph-mon[29577]: osdmap e623: 8 total, 8 up, 8 in 2024-09-10T19:15:39.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:39 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:39.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:39 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-126"}]': finished 2024-09-10T19:15:39.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:39 smithi119 ceph-mon[40674]: osdmap e623: 8 total, 8 up, 8 in 2024-09-10T19:15:40.305 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-10T19:15:40.305 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:15:40.305 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9080 ms) 2024-09-10T19:15:40.306 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-10T19:15:40.306 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:15:40.306 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6040 ms) 2024-09-10T19:15:40.306 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-10T19:15:40.306 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:15:40.306 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5037 ms) 2024-09-10T19:15:40.306 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-10T19:15:40.306 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:15:40.306 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25347 ms) 2024-09-10T19:15:40.306 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-10T19:15:40.306 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:15:40.306 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14048 ms) 2024-09-10T19:15:40.307 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-10T19:15:40.307 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12069 ms) 2024-09-10T19:15:40.307 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-10T19:15:40.307 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-10T19:15:40.307 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-10T19:15:40.307 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: 00000012 2024-09-10T19:15:40.307 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12054 ms) 2024-09-10T19:15:40.307 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-10T19:15:40.307 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: require_osd_release = quincy 2024-09-10T19:15:40.307 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6047 ms) 2024-09-10T19:15:40.307 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (554365 ms total) 2024-09-10T19:15:40.307 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: 2024-09-10T19:15:40.307 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-10T19:15:40.307 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-10T19:15:40.308 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1017 ms) 2024-09-10T19:15:40.308 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-10T19:15:40.308 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11042 ms) 2024-09-10T19:15:40.308 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-10T19:15:40.308 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18110 ms) 2024-09-10T19:15:40.308 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-10T19:15:40.308 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-09-10T19:15:40.308 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30171 ms total) 2024-09-10T19:15:40.308 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: 2024-09-10T19:15:40.308 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-10T19:15:40.308 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-10T19:15:40.308 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7028 ms) 2024-09-10T19:15:40.308 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-10T19:15:40.309 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8055 ms) 2024-09-10T19:15:40.309 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-10T19:15:40.309 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: waiting for scrub... 2024-09-10T19:15:40.309 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: done waiting 2024-09-10T19:15:40.309 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24475 ms) 2024-09-10T19:15:40.309 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-10T19:15:40.309 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9973 ms) 2024-09-10T19:15:40.309 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-10T19:15:40.309 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (7980 ms) 2024-09-10T19:15:40.309 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-10T19:15:40.309 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8055 ms) 2024-09-10T19:15:40.309 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-10T19:15:40.309 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10049 ms) 2024-09-10T19:15:40.309 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-10T19:15:40.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:15:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:15:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:15:40.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:40 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:40 smithi119 ceph-mon[29577]: pgmap v965: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:15:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:40 smithi119 ceph-mon[29577]: osdmap e624: 8 total, 8 up, 8 in 2024-09-10T19:15:40.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:40 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:40.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:40 smithi119 ceph-mon[40674]: pgmap v965: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:15:40.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:40 smithi119 ceph-mon[40674]: osdmap e624: 8 total, 8 up, 8 in 2024-09-10T19:15:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:40 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:40 smithi194 ceph-mon[32464]: pgmap v965: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-09-10T19:15:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:40 smithi194 ceph-mon[32464]: osdmap e624: 8 total, 8 up, 8 in 2024-09-10T19:15:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:42 smithi119 ceph-mon[29577]: pgmap v968: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:42 smithi119 ceph-mon[29577]: osdmap e625: 8 total, 8 up, 8 in 2024-09-10T19:15:42.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:42 smithi119 ceph-mon[40674]: pgmap v968: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:42 smithi119 ceph-mon[40674]: osdmap e625: 8 total, 8 up, 8 in 2024-09-10T19:15:42.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:42.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:42 smithi194 ceph-mon[32464]: pgmap v968: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:15:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:42 smithi194 ceph-mon[32464]: osdmap e625: 8 total, 8 up, 8 in 2024-09-10T19:15:42.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:43.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:15:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:15:42] "GET /metrics HTTP/1.1" 200 228593 "" "Prometheus/2.43.0" 2024-09-10T19:15:43.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:43 smithi194 ceph-mon[32464]: osdmap e626: 8 total, 8 up, 8 in 2024-09-10T19:15:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-128", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:43 smithi194 ceph-mon[32464]: osdmap e627: 8 total, 8 up, 8 in 2024-09-10T19:15:43.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:43 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-128"}]: dispatch 2024-09-10T19:15:43.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:43.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:43 smithi119 ceph-mon[29577]: osdmap e626: 8 total, 8 up, 8 in 2024-09-10T19:15:43.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:43.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-128", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:43.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:43 smithi119 ceph-mon[29577]: osdmap e627: 8 total, 8 up, 8 in 2024-09-10T19:15:43.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:43 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-128"}]: dispatch 2024-09-10T19:15:43.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:43.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:43 smithi119 ceph-mon[40674]: osdmap e626: 8 total, 8 up, 8 in 2024-09-10T19:15:43.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:43.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-128", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:43.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:43 smithi119 ceph-mon[40674]: osdmap e627: 8 total, 8 up, 8 in 2024-09-10T19:15:43.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:43 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-128"}]: dispatch 2024-09-10T19:15:44.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:44 smithi194 ceph-mon[32464]: pgmap v971: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:15:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-128"}]': finished 2024-09-10T19:15:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:44 smithi194 ceph-mon[32464]: osdmap e628: 8 total, 8 up, 8 in 2024-09-10T19:15:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-128", "mode": "writeback"}]: dispatch 2024-09-10T19:15:44.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:44 smithi119 ceph-mon[29577]: pgmap v971: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:15:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-128"}]': finished 2024-09-10T19:15:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:44 smithi119 ceph-mon[29577]: osdmap e628: 8 total, 8 up, 8 in 2024-09-10T19:15:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-128", "mode": "writeback"}]: dispatch 2024-09-10T19:15:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:44 smithi119 ceph-mon[40674]: pgmap v971: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-09-10T19:15:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-128"}]': finished 2024-09-10T19:15:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:44 smithi119 ceph-mon[40674]: osdmap e628: 8 total, 8 up, 8 in 2024-09-10T19:15:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-128", "mode": "writeback"}]: dispatch 2024-09-10T19:15:45.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:45 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:45.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-128", "mode": "writeback"}]': finished 2024-09-10T19:15:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:45 smithi194 ceph-mon[32464]: osdmap e629: 8 total, 8 up, 8 in 2024-09-10T19:15:45.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:45 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-128", "mode": "writeback"}]': finished 2024-09-10T19:15:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:45 smithi119 ceph-mon[29577]: osdmap e629: 8 total, 8 up, 8 in 2024-09-10T19:15:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:45 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-128", "mode": "writeback"}]': finished 2024-09-10T19:15:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:45 smithi119 ceph-mon[40674]: osdmap e629: 8 total, 8 up, 8 in 2024-09-10T19:15:46.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:46 smithi194 ceph-mon[32464]: pgmap v974: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:15:46.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:46 smithi119 ceph-mon[29577]: pgmap v974: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:15:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:46 smithi119 ceph-mon[40674]: pgmap v974: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:15:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:48 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:15:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:48 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:48 smithi194 ceph-mon[32464]: pgmap v976: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:15:48.566 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:48 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:15:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:48 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:48 smithi119 ceph-mon[29577]: pgmap v976: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:15:48.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:48 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:15:48.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:48 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:48.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:48 smithi119 ceph-mon[40674]: pgmap v976: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:15:49.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:15:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:15:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:50 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:50 smithi194 ceph-mon[32464]: pgmap v977: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-09-10T19:15:50.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:15:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:15:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:15:50.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:50 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:50 smithi119 ceph-mon[29577]: pgmap v977: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-09-10T19:15:50.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:50 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:50.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:50 smithi119 ceph-mon[40674]: pgmap v977: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 341 B/s wr, 1 op/s 2024-09-10T19:15:51.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:51.718 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:51 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:15:51.718 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:51.718 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:51 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:15:51.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:51.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:51 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:15:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-mon[29577]: pgmap v978: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 770 B/s rd, 256 B/s wr, 1 op/s 2024-09-10T19:15:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-mon[29577]: osdmap e630: 8 total, 8 up, 8 in 2024-09-10T19:15:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-128"}]: dispatch 2024-09-10T19:15:52.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:15:52.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:15:52.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:15:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-mon[40674]: pgmap v978: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 770 B/s rd, 256 B/s wr, 1 op/s 2024-09-10T19:15:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-mon[40674]: osdmap e630: 8 total, 8 up, 8 in 2024-09-10T19:15:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-128"}]: dispatch 2024-09-10T19:15:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:15:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:15:52.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:15:52.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:52 smithi194 ceph-mon[32464]: pgmap v978: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 770 B/s rd, 256 B/s wr, 1 op/s 2024-09-10T19:15:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:15:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:52 smithi194 ceph-mon[32464]: osdmap e630: 8 total, 8 up, 8 in 2024-09-10T19:15:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-128"}]: dispatch 2024-09-10T19:15:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:15:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:15:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:15:53.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:15:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:15:52] "GET /metrics HTTP/1.1" 200 228596 "" "Prometheus/2.43.0" 2024-09-10T19:15:53.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:53 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-128"}]': finished 2024-09-10T19:15:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:53 smithi194 ceph-mon[32464]: osdmap e631: 8 total, 8 up, 8 in 2024-09-10T19:15:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:53 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-128"}]': finished 2024-09-10T19:15:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:53 smithi119 ceph-mon[40674]: osdmap e631: 8 total, 8 up, 8 in 2024-09-10T19:15:53.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:53 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:15:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-128"}]': finished 2024-09-10T19:15:53.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:53 smithi119 ceph-mon[29577]: osdmap e631: 8 total, 8 up, 8 in 2024-09-10T19:15:54.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:54 smithi194 ceph-mon[32464]: pgmap v981: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 257 B/s wr, 1 op/s 2024-09-10T19:15:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:54 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:15:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:54 smithi194 ceph-mon[32464]: osdmap e632: 8 total, 8 up, 8 in 2024-09-10T19:15:54.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:54 smithi119 ceph-mon[29577]: pgmap v981: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 257 B/s wr, 1 op/s 2024-09-10T19:15:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:54 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:15:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:54 smithi119 ceph-mon[29577]: osdmap e632: 8 total, 8 up, 8 in 2024-09-10T19:15:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:54 smithi119 ceph-mon[40674]: pgmap v981: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 257 B/s wr, 1 op/s 2024-09-10T19:15:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:54 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:15:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:54 smithi119 ceph-mon[40674]: osdmap e632: 8 total, 8 up, 8 in 2024-09-10T19:15:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:55 smithi194 ceph-mon[32464]: osdmap e633: 8 total, 8 up, 8 in 2024-09-10T19:15:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:55 smithi119 ceph-mon[29577]: osdmap e633: 8 total, 8 up, 8 in 2024-09-10T19:15:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:55.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:55 smithi119 ceph-mon[40674]: osdmap e633: 8 total, 8 up, 8 in 2024-09-10T19:15:55.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:15:56.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:56 smithi194 ceph-mon[32464]: pgmap v984: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:15:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:56 smithi194 ceph-mon[32464]: osdmap e634: 8 total, 8 up, 8 in 2024-09-10T19:15:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:56 smithi119 ceph-mon[29577]: pgmap v984: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:15:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:56 smithi119 ceph-mon[29577]: osdmap e634: 8 total, 8 up, 8 in 2024-09-10T19:15:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:56 smithi119 ceph-mon[40674]: pgmap v984: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:15:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:15:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:56 smithi119 ceph-mon[40674]: osdmap e634: 8 total, 8 up, 8 in 2024-09-10T19:15:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:15:57.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-130", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:57 smithi194 ceph-mon[32464]: osdmap e635: 8 total, 8 up, 8 in 2024-09-10T19:15:57.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:57 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-130"}]: dispatch 2024-09-10T19:15:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-130", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:57 smithi119 ceph-mon[29577]: osdmap e635: 8 total, 8 up, 8 in 2024-09-10T19:15:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:57 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-130"}]: dispatch 2024-09-10T19:15:57.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-130", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:15:57.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:57 smithi119 ceph-mon[40674]: osdmap e635: 8 total, 8 up, 8 in 2024-09-10T19:15:57.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:57 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-130"}]: dispatch 2024-09-10T19:15:58.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:58 smithi194 ceph-mon[32464]: pgmap v987: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:58.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-130"}]': finished 2024-09-10T19:15:58.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:58 smithi194 ceph-mon[32464]: osdmap e636: 8 total, 8 up, 8 in 2024-09-10T19:15:58.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-130", "mode": "writeback"}]: dispatch 2024-09-10T19:15:58.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:58 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:58.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-130", "mode": "writeback"}]': finished 2024-09-10T19:15:58.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:58 smithi194 ceph-mon[32464]: osdmap e637: 8 total, 8 up, 8 in 2024-09-10T19:15:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:58 smithi119 ceph-mon[29577]: pgmap v987: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-130"}]': finished 2024-09-10T19:15:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:58 smithi119 ceph-mon[29577]: osdmap e636: 8 total, 8 up, 8 in 2024-09-10T19:15:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-130", "mode": "writeback"}]: dispatch 2024-09-10T19:15:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:58 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-130", "mode": "writeback"}]': finished 2024-09-10T19:15:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:58 smithi119 ceph-mon[29577]: osdmap e637: 8 total, 8 up, 8 in 2024-09-10T19:15:58.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:58 smithi119 ceph-mon[40674]: pgmap v987: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:15:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-130"}]': finished 2024-09-10T19:15:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:58 smithi119 ceph-mon[40674]: osdmap e636: 8 total, 8 up, 8 in 2024-09-10T19:15:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-130", "mode": "writeback"}]: dispatch 2024-09-10T19:15:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:58 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:15:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-130", "mode": "writeback"}]': finished 2024-09-10T19:15:58.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:58 smithi119 ceph-mon[40674]: osdmap e637: 8 total, 8 up, 8 in 2024-09-10T19:15:59.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:15:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:15:59.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:15:59 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:59.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:15:59 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:15:59.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:15:59 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:00.399 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:16:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:16:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:16:00.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:00 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:00 smithi194 ceph-mon[32464]: pgmap v990: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:00 smithi194 ceph-mon[32464]: osdmap e638: 8 total, 8 up, 8 in 2024-09-10T19:16:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-130"}]: dispatch 2024-09-10T19:16:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:00 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:00 smithi119 ceph-mon[29577]: pgmap v990: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:00 smithi119 ceph-mon[29577]: osdmap e638: 8 total, 8 up, 8 in 2024-09-10T19:16:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-130"}]: dispatch 2024-09-10T19:16:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:00 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:00 smithi119 ceph-mon[40674]: pgmap v990: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:00 smithi119 ceph-mon[40674]: osdmap e638: 8 total, 8 up, 8 in 2024-09-10T19:16:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-130"}]: dispatch 2024-09-10T19:16:01.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:01 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:16:01.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-130"}]': finished 2024-09-10T19:16:01.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:01 smithi194 ceph-mon[32464]: osdmap e639: 8 total, 8 up, 8 in 2024-09-10T19:16:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:01 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:16:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-130"}]': finished 2024-09-10T19:16:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:01 smithi119 ceph-mon[29577]: osdmap e639: 8 total, 8 up, 8 in 2024-09-10T19:16:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:01 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:16:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-130"}]': finished 2024-09-10T19:16:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:01 smithi119 ceph-mon[40674]: osdmap e639: 8 total, 8 up, 8 in 2024-09-10T19:16:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:02 smithi119 ceph-mon[29577]: pgmap v993: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:02 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:02 smithi119 ceph-mon[29577]: osdmap e640: 8 total, 8 up, 8 in 2024-09-10T19:16:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:16:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:02 smithi119 ceph-mon[40674]: pgmap v993: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:02 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:02 smithi119 ceph-mon[40674]: osdmap e640: 8 total, 8 up, 8 in 2024-09-10T19:16:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:16:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:02.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:02 smithi194 ceph-mon[32464]: pgmap v993: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:02 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:02 smithi194 ceph-mon[32464]: osdmap e640: 8 total, 8 up, 8 in 2024-09-10T19:16:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:16:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:03.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:16:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:16:02] "GET /metrics HTTP/1.1" 200 226160 "" "Prometheus/2.43.0" 2024-09-10T19:16:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:03 smithi194 ceph-mon[32464]: osdmap e641: 8 total, 8 up, 8 in 2024-09-10T19:16:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:03 smithi194 ceph-mon[32464]: osdmap e642: 8 total, 8 up, 8 in 2024-09-10T19:16:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:03 smithi119 ceph-mon[29577]: osdmap e641: 8 total, 8 up, 8 in 2024-09-10T19:16:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:03 smithi119 ceph-mon[29577]: osdmap e642: 8 total, 8 up, 8 in 2024-09-10T19:16:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:03 smithi119 ceph-mon[40674]: osdmap e641: 8 total, 8 up, 8 in 2024-09-10T19:16:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:03.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:03 smithi119 ceph-mon[40674]: osdmap e642: 8 total, 8 up, 8 in 2024-09-10T19:16:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:04 smithi194 ceph-mon[32464]: pgmap v996: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:16:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-132", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:04 smithi194 ceph-mon[32464]: osdmap e643: 8 total, 8 up, 8 in 2024-09-10T19:16:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-132"}]: dispatch 2024-09-10T19:16:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:04 smithi119 ceph-mon[29577]: pgmap v996: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:16:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-132", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:04 smithi119 ceph-mon[29577]: osdmap e643: 8 total, 8 up, 8 in 2024-09-10T19:16:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-132"}]: dispatch 2024-09-10T19:16:04.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:04 smithi119 ceph-mon[40674]: pgmap v996: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:16:04.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:04.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-132", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:04.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:04 smithi119 ceph-mon[40674]: osdmap e643: 8 total, 8 up, 8 in 2024-09-10T19:16:04.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-132"}]: dispatch 2024-09-10T19:16:06.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:06 smithi194 ceph-mon[32464]: pgmap v999: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:16:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-132"}]': finished 2024-09-10T19:16:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:06 smithi194 ceph-mon[32464]: osdmap e644: 8 total, 8 up, 8 in 2024-09-10T19:16:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-132", "mode": "writeback"}]: dispatch 2024-09-10T19:16:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:06 smithi119 ceph-mon[29577]: pgmap v999: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:16:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-132"}]': finished 2024-09-10T19:16:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:06 smithi119 ceph-mon[29577]: osdmap e644: 8 total, 8 up, 8 in 2024-09-10T19:16:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-132", "mode": "writeback"}]: dispatch 2024-09-10T19:16:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:06 smithi119 ceph-mon[40674]: pgmap v999: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:16:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-132"}]': finished 2024-09-10T19:16:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:06 smithi119 ceph-mon[40674]: osdmap e644: 8 total, 8 up, 8 in 2024-09-10T19:16:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-132", "mode": "writeback"}]: dispatch 2024-09-10T19:16:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:07 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:16:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-132", "mode": "writeback"}]': finished 2024-09-10T19:16:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:07 smithi194 ceph-mon[32464]: osdmap e645: 8 total, 8 up, 8 in 2024-09-10T19:16:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:07 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:07 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:16:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-132", "mode": "writeback"}]': finished 2024-09-10T19:16:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:07 smithi119 ceph-mon[29577]: osdmap e645: 8 total, 8 up, 8 in 2024-09-10T19:16:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:07 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:07 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:16:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-132", "mode": "writeback"}]': finished 2024-09-10T19:16:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:07 smithi119 ceph-mon[40674]: osdmap e645: 8 total, 8 up, 8 in 2024-09-10T19:16:07.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:07 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:08.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:08 smithi194 ceph-mon[32464]: pgmap v1002: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:08 smithi194 ceph-mon[32464]: osdmap e646: 8 total, 8 up, 8 in 2024-09-10T19:16:08.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:08 smithi119 ceph-mon[29577]: pgmap v1002: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:08 smithi119 ceph-mon[29577]: osdmap e646: 8 total, 8 up, 8 in 2024-09-10T19:16:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:08 smithi119 ceph-mon[40674]: pgmap v1002: 204 pgs: 32 creating+peering, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:08 smithi119 ceph-mon[40674]: osdmap e646: 8 total, 8 up, 8 in 2024-09-10T19:16:09.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:16:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:16:09.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:09 smithi194 ceph-mon[32464]: osdmap e647: 8 total, 8 up, 8 in 2024-09-10T19:16:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:09 smithi119 ceph-mon[29577]: osdmap e647: 8 total, 8 up, 8 in 2024-09-10T19:16:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:09 smithi119 ceph-mon[40674]: osdmap e647: 8 total, 8 up, 8 in 2024-09-10T19:16:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:10.453 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:16:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:16:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:16:10.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:10 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:10 smithi194 ceph-mon[32464]: pgmap v1005: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:16:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:10 smithi194 ceph-mon[32464]: osdmap e648: 8 total, 8 up, 8 in 2024-09-10T19:16:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-132"}]: dispatch 2024-09-10T19:16:10.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:10 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:10.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:10 smithi119 ceph-mon[29577]: pgmap v1005: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:16:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:10 smithi119 ceph-mon[29577]: osdmap e648: 8 total, 8 up, 8 in 2024-09-10T19:16:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-132"}]: dispatch 2024-09-10T19:16:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:10 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:10 smithi119 ceph-mon[40674]: pgmap v1005: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:16:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:10 smithi119 ceph-mon[40674]: osdmap e648: 8 total, 8 up, 8 in 2024-09-10T19:16:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-132"}]: dispatch 2024-09-10T19:16:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:11 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-132"}]': finished 2024-09-10T19:16:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:11 smithi194 ceph-mon[32464]: osdmap e649: 8 total, 8 up, 8 in 2024-09-10T19:16:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:11 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-132"}]': finished 2024-09-10T19:16:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:11 smithi119 ceph-mon[29577]: osdmap e649: 8 total, 8 up, 8 in 2024-09-10T19:16:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:11 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-132"}]': finished 2024-09-10T19:16:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:11 smithi119 ceph-mon[40674]: osdmap e649: 8 total, 8 up, 8 in 2024-09-10T19:16:12.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:12 smithi194 ceph-mon[32464]: pgmap v1008: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:16:12.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:12 smithi194 ceph-mon[32464]: osdmap e650: 8 total, 8 up, 8 in 2024-09-10T19:16:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:12.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:16:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:16:12] "GET /metrics HTTP/1.1" 200 226160 "" "Prometheus/2.43.0" 2024-09-10T19:16:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:12 smithi119 ceph-mon[29577]: pgmap v1008: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:16:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:12 smithi119 ceph-mon[29577]: osdmap e650: 8 total, 8 up, 8 in 2024-09-10T19:16:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:12.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:12 smithi119 ceph-mon[40674]: pgmap v1008: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:16:12.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:12 smithi119 ceph-mon[40674]: osdmap e650: 8 total, 8 up, 8 in 2024-09-10T19:16:12.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:13.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:13 smithi194 ceph-mon[32464]: osdmap e651: 8 total, 8 up, 8 in 2024-09-10T19:16:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-132"}]: dispatch 2024-09-10T19:16:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:13 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:16:13.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:13 smithi119 ceph-mon[29577]: osdmap e651: 8 total, 8 up, 8 in 2024-09-10T19:16:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-132"}]: dispatch 2024-09-10T19:16:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:13 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:16:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:13 smithi119 ceph-mon[40674]: osdmap e651: 8 total, 8 up, 8 in 2024-09-10T19:16:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-132"}]: dispatch 2024-09-10T19:16:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:13 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:16:15.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:14 smithi194 ceph-mon[32464]: pgmap v1011: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T19:16:15.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-132"}]': finished 2024-09-10T19:16:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:14 smithi194 ceph-mon[32464]: osdmap e652: 8 total, 8 up, 8 in 2024-09-10T19:16:15.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:14 smithi119 ceph-mon[29577]: pgmap v1011: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T19:16:15.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-132"}]': finished 2024-09-10T19:16:15.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:14 smithi119 ceph-mon[29577]: osdmap e652: 8 total, 8 up, 8 in 2024-09-10T19:16:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:14 smithi119 ceph-mon[40674]: pgmap v1011: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-09-10T19:16:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-132"}]': finished 2024-09-10T19:16:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:14 smithi119 ceph-mon[40674]: osdmap e652: 8 total, 8 up, 8 in 2024-09-10T19:16:16.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:15 smithi194 ceph-mon[32464]: osdmap e653: 8 total, 8 up, 8 in 2024-09-10T19:16:16.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:15 smithi194 ceph-mon[32464]: pgmap v1014: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:16:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:15 smithi194 ceph-mon[32464]: osdmap e654: 8 total, 8 up, 8 in 2024-09-10T19:16:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:16.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:15 smithi119 ceph-mon[29577]: osdmap e653: 8 total, 8 up, 8 in 2024-09-10T19:16:16.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:15 smithi119 ceph-mon[29577]: pgmap v1014: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:16:16.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:15 smithi119 ceph-mon[29577]: osdmap e654: 8 total, 8 up, 8 in 2024-09-10T19:16:16.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:16.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:15 smithi119 ceph-mon[40674]: osdmap e653: 8 total, 8 up, 8 in 2024-09-10T19:16:16.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:15 smithi119 ceph-mon[40674]: pgmap v1014: 172 pgs: 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-09-10T19:16:16.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:15 smithi119 ceph-mon[40674]: osdmap e654: 8 total, 8 up, 8 in 2024-09-10T19:16:16.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:17.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:17.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:17 smithi194 ceph-mon[32464]: osdmap e655: 8 total, 8 up, 8 in 2024-09-10T19:16:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-134", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:17 smithi194 ceph-mon[32464]: osdmap e656: 8 total, 8 up, 8 in 2024-09-10T19:16:17.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-134"}]: dispatch 2024-09-10T19:16:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:17 smithi119 ceph-mon[29577]: osdmap e655: 8 total, 8 up, 8 in 2024-09-10T19:16:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-134", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:17 smithi119 ceph-mon[29577]: osdmap e656: 8 total, 8 up, 8 in 2024-09-10T19:16:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-134"}]: dispatch 2024-09-10T19:16:17.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:17.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:17 smithi119 ceph-mon[40674]: osdmap e655: 8 total, 8 up, 8 in 2024-09-10T19:16:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-134", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:17 smithi119 ceph-mon[40674]: osdmap e656: 8 total, 8 up, 8 in 2024-09-10T19:16:17.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-134"}]: dispatch 2024-09-10T19:16:18.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:18 smithi194 ceph-mon[32464]: pgmap v1017: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:18.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-134"}]': finished 2024-09-10T19:16:18.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:18 smithi194 ceph-mon[32464]: osdmap e657: 8 total, 8 up, 8 in 2024-09-10T19:16:18.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-134", "mode": "writeback"}]: dispatch 2024-09-10T19:16:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:18 smithi119 ceph-mon[29577]: pgmap v1017: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-134"}]': finished 2024-09-10T19:16:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:18 smithi119 ceph-mon[29577]: osdmap e657: 8 total, 8 up, 8 in 2024-09-10T19:16:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-134", "mode": "writeback"}]: dispatch 2024-09-10T19:16:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:18 smithi119 ceph-mon[40674]: pgmap v1017: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-134"}]': finished 2024-09-10T19:16:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:18 smithi119 ceph-mon[40674]: osdmap e657: 8 total, 8 up, 8 in 2024-09-10T19:16:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-134", "mode": "writeback"}]: dispatch 2024-09-10T19:16:19.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:16:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:16:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:19 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.a", "id": [1, 3]}]: dispatch 2024-09-10T19:16:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:19 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.10", "id": [1, 3]}]: dispatch 2024-09-10T19:16:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:19 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.18", "id": [1, 3]}]: dispatch 2024-09-10T19:16:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:19 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:16:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-134", "mode": "writeback"}]': finished 2024-09-10T19:16:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:19 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.a", "id": [1, 3]}]': finished 2024-09-10T19:16:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:19 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.10", "id": [1, 3]}]': finished 2024-09-10T19:16:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:19 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.18", "id": [1, 3]}]': finished 2024-09-10T19:16:19.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:19 smithi194 ceph-mon[32464]: osdmap e658: 8 total, 8 up, 8 in 2024-09-10T19:16:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.a", "id": [1, 3]}]: dispatch 2024-09-10T19:16:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.10", "id": [1, 3]}]: dispatch 2024-09-10T19:16:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.18", "id": [1, 3]}]: dispatch 2024-09-10T19:16:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:16:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-134", "mode": "writeback"}]': finished 2024-09-10T19:16:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.a", "id": [1, 3]}]': finished 2024-09-10T19:16:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.10", "id": [1, 3]}]': finished 2024-09-10T19:16:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.18", "id": [1, 3]}]': finished 2024-09-10T19:16:19.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[29577]: osdmap e658: 8 total, 8 up, 8 in 2024-09-10T19:16:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.a", "id": [1, 3]}]: dispatch 2024-09-10T19:16:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.10", "id": [1, 3]}]: dispatch 2024-09-10T19:16:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.18", "id": [1, 3]}]: dispatch 2024-09-10T19:16:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:16:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-134", "mode": "writeback"}]': finished 2024-09-10T19:16:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.a", "id": [1, 3]}]': finished 2024-09-10T19:16:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.10", "id": [1, 3]}]': finished 2024-09-10T19:16:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "253.18", "id": [1, 3]}]': finished 2024-09-10T19:16:19.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:19 smithi119 ceph-mon[40674]: osdmap e658: 8 total, 8 up, 8 in 2024-09-10T19:16:20.502 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:16:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:16:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:16:20.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:20 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:20.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:20 smithi194 ceph-mon[32464]: pgmap v1020: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:20.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:20 smithi194 ceph-mon[32464]: osdmap e659: 8 total, 8 up, 8 in 2024-09-10T19:16:20.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:20.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:20 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:20 smithi119 ceph-mon[29577]: pgmap v1020: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:20 smithi119 ceph-mon[29577]: osdmap e659: 8 total, 8 up, 8 in 2024-09-10T19:16:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:20 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:20 smithi119 ceph-mon[40674]: pgmap v1020: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:20 smithi119 ceph-mon[40674]: osdmap e659: 8 total, 8 up, 8 in 2024-09-10T19:16:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:22.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:22 smithi119 ceph-mon[29577]: pgmap v1023: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:22.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:22.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:22 smithi119 ceph-mon[29577]: osdmap e660: 8 total, 8 up, 8 in 2024-09-10T19:16:22.619 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:22 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-134"}]: dispatch 2024-09-10T19:16:22.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:22 smithi119 ceph-mon[40674]: pgmap v1023: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:22.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:22.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:22 smithi119 ceph-mon[40674]: osdmap e660: 8 total, 8 up, 8 in 2024-09-10T19:16:22.620 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:22 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-134"}]: dispatch 2024-09-10T19:16:22.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:22 smithi194 ceph-mon[32464]: pgmap v1023: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:22 smithi194 ceph-mon[32464]: osdmap e660: 8 total, 8 up, 8 in 2024-09-10T19:16:22.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:22 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-134"}]: dispatch 2024-09-10T19:16:23.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:16:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:16:22] "GET /metrics HTTP/1.1" 200 228615 "" "Prometheus/2.43.0" 2024-09-10T19:16:23.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:23 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:16:23.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:23 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-134"}]': finished 2024-09-10T19:16:23.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:23 smithi194 ceph-mon[32464]: osdmap e661: 8 total, 8 up, 8 in 2024-09-10T19:16:23.816 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:23 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:16:23.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:23 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-134"}]': finished 2024-09-10T19:16:23.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:23 smithi119 ceph-mon[40674]: osdmap e661: 8 total, 8 up, 8 in 2024-09-10T19:16:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:23 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:16:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:23 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-134"}]': finished 2024-09-10T19:16:23.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:23 smithi119 ceph-mon[29577]: osdmap e661: 8 total, 8 up, 8 in 2024-09-10T19:16:24.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:24 smithi194 ceph-mon[32464]: pgmap v1026: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-09-10T19:16:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:24 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:24.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:24 smithi194 ceph-mon[32464]: osdmap e662: 8 total, 8 up, 8 in 2024-09-10T19:16:24.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:24 smithi119 ceph-mon[40674]: pgmap v1026: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-09-10T19:16:24.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:24 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:24.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:24 smithi119 ceph-mon[40674]: osdmap e662: 8 total, 8 up, 8 in 2024-09-10T19:16:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:24 smithi119 ceph-mon[29577]: pgmap v1026: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.2 KiB/s wr, 3 op/s 2024-09-10T19:16:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:24 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:24 smithi119 ceph-mon[29577]: osdmap e662: 8 total, 8 up, 8 in 2024-09-10T19:16:25.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:25 smithi194 ceph-mon[32464]: osdmap e663: 8 total, 8 up, 8 in 2024-09-10T19:16:25.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:25 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:25.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:25 smithi119 ceph-mon[29577]: osdmap e663: 8 total, 8 up, 8 in 2024-09-10T19:16:25.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:25 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:25.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:25 smithi119 ceph-mon[40674]: osdmap e663: 8 total, 8 up, 8 in 2024-09-10T19:16:25.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:25 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:26 smithi194 ceph-mon[32464]: pgmap v1029: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:16:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:26 smithi194 ceph-mon[32464]: osdmap e664: 8 total, 8 up, 8 in 2024-09-10T19:16:26.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:26 smithi119 ceph-mon[29577]: pgmap v1029: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:16:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:26 smithi119 ceph-mon[29577]: osdmap e664: 8 total, 8 up, 8 in 2024-09-10T19:16:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:26 smithi119 ceph-mon[40674]: pgmap v1029: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-09-10T19:16:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:26.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:26 smithi119 ceph-mon[40674]: osdmap e664: 8 total, 8 up, 8 in 2024-09-10T19:16:26.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:27.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-136", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:27 smithi194 ceph-mon[32464]: osdmap e665: 8 total, 8 up, 8 in 2024-09-10T19:16:27.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-136"}]: dispatch 2024-09-10T19:16:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-136", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:27 smithi119 ceph-mon[29577]: osdmap e665: 8 total, 8 up, 8 in 2024-09-10T19:16:27.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-136"}]: dispatch 2024-09-10T19:16:27.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-136", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:27.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:27 smithi119 ceph-mon[40674]: osdmap e665: 8 total, 8 up, 8 in 2024-09-10T19:16:27.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-136"}]: dispatch 2024-09-10T19:16:28.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:28 smithi194 ceph-mon[32464]: pgmap v1032: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:28.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-136"}]': finished 2024-09-10T19:16:28.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:28 smithi194 ceph-mon[32464]: osdmap e666: 8 total, 8 up, 8 in 2024-09-10T19:16:28.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-136", "mode": "writeback"}]: dispatch 2024-09-10T19:16:28.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:28 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:16:28.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-136", "mode": "writeback"}]': finished 2024-09-10T19:16:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:28 smithi119 ceph-mon[29577]: pgmap v1032: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-136"}]': finished 2024-09-10T19:16:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:28 smithi119 ceph-mon[29577]: osdmap e666: 8 total, 8 up, 8 in 2024-09-10T19:16:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-136", "mode": "writeback"}]: dispatch 2024-09-10T19:16:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:28 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:16:28.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-136", "mode": "writeback"}]': finished 2024-09-10T19:16:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:28 smithi119 ceph-mon[40674]: pgmap v1032: 204 pgs: 32 unknown, 172 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:28.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-136"}]': finished 2024-09-10T19:16:28.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:28 smithi119 ceph-mon[40674]: osdmap e666: 8 total, 8 up, 8 in 2024-09-10T19:16:28.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-136", "mode": "writeback"}]: dispatch 2024-09-10T19:16:28.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:28 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:16:28.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-136", "mode": "writeback"}]': finished 2024-09-10T19:16:29.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:16:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:16:29.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:29 smithi194 ceph-mon[32464]: osdmap e667: 8 total, 8 up, 8 in 2024-09-10T19:16:29.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:29 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:29 smithi119 ceph-mon[29577]: osdmap e667: 8 total, 8 up, 8 in 2024-09-10T19:16:29.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:29 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:29.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:29 smithi119 ceph-mon[40674]: osdmap e667: 8 total, 8 up, 8 in 2024-09-10T19:16:29.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:29 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:30.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:16:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:16:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:16:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:30 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:30 smithi194 ceph-mon[32464]: pgmap v1035: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:30 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:30 smithi194 ceph-mon[32464]: osdmap e668: 8 total, 8 up, 8 in 2024-09-10T19:16:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:30 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-136"}]: dispatch 2024-09-10T19:16:30.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:30 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:30 smithi119 ceph-mon[29577]: pgmap v1035: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:30 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:30 smithi119 ceph-mon[29577]: osdmap e668: 8 total, 8 up, 8 in 2024-09-10T19:16:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:30 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-136"}]: dispatch 2024-09-10T19:16:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:30 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:30 smithi119 ceph-mon[40674]: pgmap v1035: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:30 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:16:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:30 smithi119 ceph-mon[40674]: osdmap e668: 8 total, 8 up, 8 in 2024-09-10T19:16:30.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:30 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-136"}]: dispatch 2024-09-10T19:16:31.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:31 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:16:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-136"}]': finished 2024-09-10T19:16:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:31 smithi194 ceph-mon[32464]: osdmap e669: 8 total, 8 up, 8 in 2024-09-10T19:16:31.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:31 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:16:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-136"}]': finished 2024-09-10T19:16:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:31 smithi119 ceph-mon[29577]: osdmap e669: 8 total, 8 up, 8 in 2024-09-10T19:16:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:31 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:16:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-136"}]': finished 2024-09-10T19:16:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:31 smithi119 ceph-mon[40674]: osdmap e669: 8 total, 8 up, 8 in 2024-09-10T19:16:32.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:32 smithi119 ceph-mon[29577]: pgmap v1038: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:32.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:32 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:32.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:32 smithi119 ceph-mon[29577]: osdmap e670: 8 total, 8 up, 8 in 2024-09-10T19:16:32.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:32.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:32 smithi119 ceph-mon[40674]: pgmap v1038: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:32.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:32 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:32.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:32 smithi119 ceph-mon[40674]: osdmap e670: 8 total, 8 up, 8 in 2024-09-10T19:16:32.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:32 smithi194 ceph-mon[32464]: pgmap v1038: 204 pgs: 204 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:32 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:32 smithi194 ceph-mon[32464]: osdmap e670: 8 total, 8 up, 8 in 2024-09-10T19:16:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:33.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:16:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:16:32] "GET /metrics HTTP/1.1" 200 226160 "" "Prometheus/2.43.0" 2024-09-10T19:16:33.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:33 smithi194 ceph-mon[32464]: osdmap e671: 8 total, 8 up, 8 in 2024-09-10T19:16:33.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:33 smithi119 ceph-mon[29577]: osdmap e671: 8 total, 8 up, 8 in 2024-09-10T19:16:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:33 smithi119 ceph-mon[40674]: osdmap e671: 8 total, 8 up, 8 in 2024-09-10T19:16:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:34.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:34 smithi194 ceph-mon[32464]: pgmap v1041: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-09-10T19:16:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:34 smithi194 ceph-mon[32464]: osdmap e672: 8 total, 8 up, 8 in 2024-09-10T19:16:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:34 smithi119 ceph-mon[29577]: pgmap v1041: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-09-10T19:16:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:34 smithi119 ceph-mon[29577]: osdmap e672: 8 total, 8 up, 8 in 2024-09-10T19:16:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:34.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:34 smithi119 ceph-mon[40674]: pgmap v1041: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 977 KiB/s wr, 1 op/s 2024-09-10T19:16:34.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:34.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:34 smithi119 ceph-mon[40674]: osdmap e672: 8 total, 8 up, 8 in 2024-09-10T19:16:34.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:35.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-138", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:35 smithi194 ceph-mon[32464]: osdmap e673: 8 total, 8 up, 8 in 2024-09-10T19:16:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:16:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:16:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:35 smithi194 ceph-mon[32464]: osdmap e674: 8 total, 8 up, 8 in 2024-09-10T19:16:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:16:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-138", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:35 smithi119 ceph-mon[29577]: osdmap e673: 8 total, 8 up, 8 in 2024-09-10T19:16:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:16:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:16:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:35 smithi119 ceph-mon[29577]: osdmap e674: 8 total, 8 up, 8 in 2024-09-10T19:16:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:16:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-138", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:35 smithi119 ceph-mon[40674]: osdmap e673: 8 total, 8 up, 8 in 2024-09-10T19:16:35.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:16:35.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:16:35.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:35 smithi119 ceph-mon[40674]: osdmap e674: 8 total, 8 up, 8 in 2024-09-10T19:16:35.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:16:36.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:36 smithi194 ceph-mon[32464]: pgmap v1044: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-09-10T19:16:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:16:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:36 smithi194 ceph-mon[32464]: osdmap e675: 8 total, 8 up, 8 in 2024-09-10T19:16:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:16:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:36 smithi119 ceph-mon[29577]: pgmap v1044: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-09-10T19:16:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:16:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:36 smithi119 ceph-mon[29577]: osdmap e675: 8 total, 8 up, 8 in 2024-09-10T19:16:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:16:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:36 smithi119 ceph-mon[40674]: pgmap v1044: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-09-10T19:16:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:16:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:36 smithi119 ceph-mon[40674]: osdmap e675: 8 total, 8 up, 8 in 2024-09-10T19:16:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-09-10T19:16:37.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:37 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:16:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:37 smithi194 ceph-mon[32464]: osdmap e676: 8 total, 8 up, 8 in 2024-09-10T19:16:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:37 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:16:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:37 smithi119 ceph-mon[29577]: osdmap e676: 8 total, 8 up, 8 in 2024-09-10T19:16:37.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:37 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:37.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-09-10T19:16:37.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:37 smithi119 ceph-mon[40674]: osdmap e676: 8 total, 8 up, 8 in 2024-09-10T19:16:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:38 smithi194 ceph-mon[32464]: pgmap v1047: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:38.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:38 smithi119 ceph-mon[29577]: pgmap v1047: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:38.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:38 smithi119 ceph-mon[40674]: pgmap v1047: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:39.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:16:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:16:39.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:39 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:39.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:39 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-138"}]: dispatch 2024-09-10T19:16:39.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:39 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:39.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:39 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-138"}]: dispatch 2024-09-10T19:16:39.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:39 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:16:39.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:39 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-138"}]: dispatch 2024-09-10T19:16:40.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:16:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:16:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:16:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:40 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:40 smithi194 ceph-mon[32464]: pgmap v1049: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:16:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-138"}]': finished 2024-09-10T19:16:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:40 smithi194 ceph-mon[32464]: osdmap e677: 8 total, 8 up, 8 in 2024-09-10T19:16:41.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:40 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:40 smithi119 ceph-mon[29577]: pgmap v1049: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:16:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-138"}]': finished 2024-09-10T19:16:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:40 smithi119 ceph-mon[29577]: osdmap e677: 8 total, 8 up, 8 in 2024-09-10T19:16:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:40 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:40 smithi119 ceph-mon[40674]: pgmap v1049: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:16:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-138"}]': finished 2024-09-10T19:16:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:40 smithi119 ceph-mon[40674]: osdmap e677: 8 total, 8 up, 8 in 2024-09-10T19:16:42.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:41 smithi194 ceph-mon[32464]: osdmap e678: 8 total, 8 up, 8 in 2024-09-10T19:16:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:41 smithi194 ceph-mon[32464]: osdmap e679: 8 total, 8 up, 8 in 2024-09-10T19:16:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:42.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:41 smithi119 ceph-mon[29577]: osdmap e678: 8 total, 8 up, 8 in 2024-09-10T19:16:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:41 smithi119 ceph-mon[29577]: osdmap e679: 8 total, 8 up, 8 in 2024-09-10T19:16:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:41 smithi119 ceph-mon[40674]: osdmap e678: 8 total, 8 up, 8 in 2024-09-10T19:16:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:41 smithi119 ceph-mon[40674]: osdmap e679: 8 total, 8 up, 8 in 2024-09-10T19:16:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:43.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:42 smithi194 ceph-mon[32464]: pgmap v1052: 172 pgs: 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:16:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:42 smithi194 ceph-mon[32464]: osdmap e680: 8 total, 8 up, 8 in 2024-09-10T19:16:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:43.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:16:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:16:42] "GET /metrics HTTP/1.1" 200 226160 "" "Prometheus/2.43.0" 2024-09-10T19:16:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:42 smithi119 ceph-mon[29577]: pgmap v1052: 172 pgs: 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:16:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:42 smithi119 ceph-mon[29577]: osdmap e680: 8 total, 8 up, 8 in 2024-09-10T19:16:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:42 smithi119 ceph-mon[40674]: pgmap v1052: 172 pgs: 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:16:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:42 smithi119 ceph-mon[40674]: osdmap e680: 8 total, 8 up, 8 in 2024-09-10T19:16:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:16:44.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:44 smithi194 ceph-mon[32464]: pgmap v1055: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-140", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:44 smithi194 ceph-mon[32464]: osdmap e681: 8 total, 8 up, 8 in 2024-09-10T19:16:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:16:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:44 smithi119 ceph-mon[29577]: pgmap v1055: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-140", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:44 smithi119 ceph-mon[29577]: osdmap e681: 8 total, 8 up, 8 in 2024-09-10T19:16:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:16:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:44 smithi119 ceph-mon[40674]: pgmap v1055: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-140", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:16:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:44 smithi119 ceph-mon[40674]: osdmap e681: 8 total, 8 up, 8 in 2024-09-10T19:16:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_count","val": "3"}]: dispatch 2024-09-10T19:16:45.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_count","val": "3"}]': finished 2024-09-10T19:16:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:45 smithi194 ceph-mon[32464]: osdmap e682: 8 total, 8 up, 8 in 2024-09-10T19:16:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:16:45.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:45 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_period","val": "3"}]': finished 2024-09-10T19:16:45.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_count","val": "3"}]': finished 2024-09-10T19:16:45.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:45 smithi119 ceph-mon[29577]: osdmap e682: 8 total, 8 up, 8 in 2024-09-10T19:16:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:16:45.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:45 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_period","val": "3"}]': finished 2024-09-10T19:16:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_count","val": "3"}]': finished 2024-09-10T19:16:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:45 smithi119 ceph-mon[40674]: osdmap e682: 8 total, 8 up, 8 in 2024-09-10T19:16:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_period","val": "3"}]: dispatch 2024-09-10T19:16:45.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:45 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_period","val": "3"}]': finished 2024-09-10T19:16:46.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:46 smithi194 ceph-mon[32464]: pgmap v1058: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:46 smithi194 ceph-mon[32464]: osdmap e683: 8 total, 8 up, 8 in 2024-09-10T19:16:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:16:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:16:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:46 smithi194 ceph-mon[32464]: osdmap e684: 8 total, 8 up, 8 in 2024-09-10T19:16:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:46 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:16:46.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:46 smithi119 ceph-mon[29577]: pgmap v1058: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:46 smithi119 ceph-mon[29577]: osdmap e683: 8 total, 8 up, 8 in 2024-09-10T19:16:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:16:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:16:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:46 smithi119 ceph-mon[29577]: osdmap e684: 8 total, 8 up, 8 in 2024-09-10T19:16:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:46 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:16:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:46 smithi119 ceph-mon[40674]: pgmap v1058: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:46 smithi119 ceph-mon[40674]: osdmap e683: 8 total, 8 up, 8 in 2024-09-10T19:16:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:16:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:16:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:46 smithi119 ceph-mon[40674]: osdmap e684: 8 total, 8 up, 8 in 2024-09-10T19:16:46.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:46 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-09-10T19:16:47.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:47 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-10T19:16:47.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:47 smithi194 ceph-mon[32464]: osdmap e685: 8 total, 8 up, 8 in 2024-09-10T19:16:47.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:47 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-10T19:16:47.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:47 smithi119 ceph-mon[29577]: osdmap e685: 8 total, 8 up, 8 in 2024-09-10T19:16:47.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:47.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:47 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-09-10T19:16:47.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:47 smithi119 ceph-mon[40674]: osdmap e685: 8 total, 8 up, 8 in 2024-09-10T19:16:48.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:48 smithi194 ceph-mon[32464]: pgmap v1061: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:48.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:48 smithi119 ceph-mon[29577]: pgmap v1061: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:48.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:48 smithi119 ceph-mon[40674]: pgmap v1061: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:16:49.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:16:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:16:50.465 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:16:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:16:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:16:50.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:50 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:50.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:50 smithi194 ceph-mon[32464]: pgmap v1063: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-09-10T19:16:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:50 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:50 smithi119 ceph-mon[29577]: pgmap v1063: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-09-10T19:16:50.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:50 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:50.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:50 smithi119 ceph-mon[40674]: pgmap v1063: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-09-10T19:16:52.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:52 smithi194 ceph-mon[32464]: pgmap v1064: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 1.3 KiB/s wr, 0 op/s 2024-09-10T19:16:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:16:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:16:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:16:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:16:52.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:16:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:16:52] "GET /metrics HTTP/1.1" 200 228631 "" "Prometheus/2.43.0" 2024-09-10T19:16:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:52 smithi119 ceph-mon[29577]: pgmap v1064: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 1.3 KiB/s wr, 0 op/s 2024-09-10T19:16:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:16:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:16:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:16:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:16:52.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:52 smithi119 ceph-mon[40674]: pgmap v1064: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 1.3 KiB/s wr, 0 op/s 2024-09-10T19:16:52.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:16:52.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:16:52.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:16:52.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:16:54.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:54 smithi194 ceph-mon[32464]: pgmap v1065: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 6.2 KiB/s wr, 2 op/s 2024-09-10T19:16:54.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:54 smithi119 ceph-mon[29577]: pgmap v1065: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 6.2 KiB/s wr, 2 op/s 2024-09-10T19:16:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:54 smithi119 ceph-mon[40674]: pgmap v1065: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 6.2 KiB/s wr, 2 op/s 2024-09-10T19:16:56.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:56 smithi194 ceph-mon[32464]: pgmap v1066: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 5.5 KiB/s wr, 1 op/s 2024-09-10T19:16:56.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:56 smithi119 ceph-mon[29577]: pgmap v1066: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 5.5 KiB/s wr, 1 op/s 2024-09-10T19:16:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:56 smithi119 ceph-mon[40674]: pgmap v1066: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 5.5 KiB/s wr, 1 op/s 2024-09-10T19:16:58.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:16:58 smithi194 ceph-mon[32464]: pgmap v1067: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 4.9 KiB/s wr, 1 op/s 2024-09-10T19:16:58.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:16:58 smithi119 ceph-mon[29577]: pgmap v1067: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 4.9 KiB/s wr, 1 op/s 2024-09-10T19:16:58.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:16:58 smithi119 ceph-mon[40674]: pgmap v1067: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 4.9 KiB/s wr, 1 op/s 2024-09-10T19:16:59.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:16:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:17:00.487 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:17:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:17:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:17:00.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:00 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:00 smithi194 ceph-mon[32464]: pgmap v1068: 204 pgs: 204 active+clean; 4.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 7.7 KiB/s wr, 2 op/s 2024-09-10T19:17:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-140"}]: dispatch 2024-09-10T19:17:00.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:00 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:00 smithi119 ceph-mon[29577]: pgmap v1068: 204 pgs: 204 active+clean; 4.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 7.7 KiB/s wr, 2 op/s 2024-09-10T19:17:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-140"}]: dispatch 2024-09-10T19:17:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:00 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:00 smithi119 ceph-mon[40674]: pgmap v1068: 204 pgs: 204 active+clean; 4.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 7.7 KiB/s wr, 2 op/s 2024-09-10T19:17:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-140"}]: dispatch 2024-09-10T19:17:01.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-140"}]': finished 2024-09-10T19:17:01.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:01 smithi194 ceph-mon[32464]: osdmap e686: 8 total, 8 up, 8 in 2024-09-10T19:17:01.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-140"}]': finished 2024-09-10T19:17:01.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:01 smithi119 ceph-mon[29577]: osdmap e686: 8 total, 8 up, 8 in 2024-09-10T19:17:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-140"}]': finished 2024-09-10T19:17:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:01 smithi119 ceph-mon[40674]: osdmap e686: 8 total, 8 up, 8 in 2024-09-10T19:17:02.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:02 smithi194 ceph-mon[32464]: pgmap v1070: 204 pgs: 204 active+clean; 4.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-09-10T19:17:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:02 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:02 smithi194 ceph-mon[32464]: osdmap e687: 8 total, 8 up, 8 in 2024-09-10T19:17:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:02.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:17:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:17:02] "GET /metrics HTTP/1.1" 200 226141 "" "Prometheus/2.43.0" 2024-09-10T19:17:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:02 smithi119 ceph-mon[29577]: pgmap v1070: 204 pgs: 204 active+clean; 4.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-09-10T19:17:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:02 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:02 smithi119 ceph-mon[29577]: osdmap e687: 8 total, 8 up, 8 in 2024-09-10T19:17:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:02.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:02 smithi119 ceph-mon[40674]: pgmap v1070: 204 pgs: 204 active+clean; 4.5 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-09-10T19:17:02.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:02 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:02.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:02 smithi119 ceph-mon[40674]: osdmap e687: 8 total, 8 up, 8 in 2024-09-10T19:17:02.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:03.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:03 smithi194 ceph-mon[32464]: osdmap e688: 8 total, 8 up, 8 in 2024-09-10T19:17:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:17:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:17:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:03 smithi194 ceph-mon[32464]: osdmap e689: 8 total, 8 up, 8 in 2024-09-10T19:17:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:03 smithi119 ceph-mon[29577]: osdmap e688: 8 total, 8 up, 8 in 2024-09-10T19:17:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:17:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:17:03.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:03 smithi119 ceph-mon[29577]: osdmap e689: 8 total, 8 up, 8 in 2024-09-10T19:17:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:03 smithi119 ceph-mon[40674]: osdmap e688: 8 total, 8 up, 8 in 2024-09-10T19:17:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:17:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:17:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:03 smithi119 ceph-mon[40674]: osdmap e689: 8 total, 8 up, 8 in 2024-09-10T19:17:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:04 smithi119 ceph-mon[29577]: pgmap v1073: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:17:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:17:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:17:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:04 smithi119 ceph-mon[29577]: osdmap e690: 8 total, 8 up, 8 in 2024-09-10T19:17:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-142"}]: dispatch 2024-09-10T19:17:04.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:04 smithi119 ceph-mon[40674]: pgmap v1073: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:17:04.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:17:04.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:17:04.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:04 smithi119 ceph-mon[40674]: osdmap e690: 8 total, 8 up, 8 in 2024-09-10T19:17:04.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-142"}]: dispatch 2024-09-10T19:17:05.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:04 smithi194 ceph-mon[32464]: pgmap v1073: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:17:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:17:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:17:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:04 smithi194 ceph-mon[32464]: osdmap e690: 8 total, 8 up, 8 in 2024-09-10T19:17:05.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-142"}]: dispatch 2024-09-10T19:17:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:06 smithi119 ceph-mon[29577]: pgmap v1076: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:17:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-142"}]': finished 2024-09-10T19:17:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:06 smithi119 ceph-mon[29577]: osdmap e691: 8 total, 8 up, 8 in 2024-09-10T19:17:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-142", "mode": "writeback"}]: dispatch 2024-09-10T19:17:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:06 smithi119 ceph-mon[40674]: pgmap v1076: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:17:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-142"}]': finished 2024-09-10T19:17:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:06 smithi119 ceph-mon[40674]: osdmap e691: 8 total, 8 up, 8 in 2024-09-10T19:17:06.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-142", "mode": "writeback"}]: dispatch 2024-09-10T19:17:07.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:06 smithi194 ceph-mon[32464]: pgmap v1076: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:17:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-142"}]': finished 2024-09-10T19:17:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:06 smithi194 ceph-mon[32464]: osdmap e691: 8 total, 8 up, 8 in 2024-09-10T19:17:07.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-142", "mode": "writeback"}]: dispatch 2024-09-10T19:17:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:07 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:17:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-142", "mode": "writeback"}]': finished 2024-09-10T19:17:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:07 smithi119 ceph-mon[40674]: osdmap e692: 8 total, 8 up, 8 in 2024-09-10T19:17:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:17:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:07 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:07 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:17:07.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-142", "mode": "writeback"}]': finished 2024-09-10T19:17:07.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:07 smithi119 ceph-mon[29577]: osdmap e692: 8 total, 8 up, 8 in 2024-09-10T19:17:07.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:17:07.818 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:07 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:07 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:17:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-142", "mode": "writeback"}]': finished 2024-09-10T19:17:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:07 smithi194 ceph-mon[32464]: osdmap e692: 8 total, 8 up, 8 in 2024-09-10T19:17:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:17:08.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:07 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:08 smithi119 ceph-mon[29577]: pgmap v1079: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:17:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:17:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:08 smithi119 ceph-mon[29577]: osdmap e693: 8 total, 8 up, 8 in 2024-09-10T19:17:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:17:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:17:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:08 smithi119 ceph-mon[29577]: osdmap e694: 8 total, 8 up, 8 in 2024-09-10T19:17:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:17:08.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:08 smithi119 ceph-mon[40674]: pgmap v1079: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:17:08.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:17:08.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:08 smithi119 ceph-mon[40674]: osdmap e693: 8 total, 8 up, 8 in 2024-09-10T19:17:08.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:17:08.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:17:08.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:08 smithi119 ceph-mon[40674]: osdmap e694: 8 total, 8 up, 8 in 2024-09-10T19:17:08.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:17:08.868 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:08 smithi194 ceph-mon[32464]: pgmap v1079: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:17:08.868 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:17:08.868 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:08 smithi194 ceph-mon[32464]: osdmap e693: 8 total, 8 up, 8 in 2024-09-10T19:17:08.868 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:17:08.868 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:17:08.868 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:08 smithi194 ceph-mon[32464]: osdmap e694: 8 total, 8 up, 8 in 2024-09-10T19:17:08.868 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:17:09.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:17:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:17:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:09 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:17:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:17:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:09 smithi119 ceph-mon[29577]: osdmap e695: 8 total, 8 up, 8 in 2024-09-10T19:17:09.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:17:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:09 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:17:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:17:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:09 smithi119 ceph-mon[40674]: osdmap e695: 8 total, 8 up, 8 in 2024-09-10T19:17:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:17:10.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:09 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:17:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:17:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:09 smithi194 ceph-mon[32464]: osdmap e695: 8 total, 8 up, 8 in 2024-09-10T19:17:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:17:10.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:17:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:17:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:17:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:10 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:10 smithi119 ceph-mon[29577]: pgmap v1082: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T19:17:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:17:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:10 smithi119 ceph-mon[29577]: osdmap e696: 8 total, 8 up, 8 in 2024-09-10T19:17:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:17:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:10 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:10 smithi119 ceph-mon[40674]: pgmap v1082: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T19:17:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:17:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:10 smithi119 ceph-mon[40674]: osdmap e696: 8 total, 8 up, 8 in 2024-09-10T19:17:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:17:11.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:10 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:11.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:10 smithi194 ceph-mon[32464]: pgmap v1082: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T19:17:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:17:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:10 smithi194 ceph-mon[32464]: osdmap e696: 8 total, 8 up, 8 in 2024-09-10T19:17:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-09-10T19:17:12.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:17:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:17:12] "GET /metrics HTTP/1.1" 200 226141 "" "Prometheus/2.43.0" 2024-09-10T19:17:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:12 smithi119 ceph-mon[29577]: pgmap v1085: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T19:17:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-10T19:17:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:12 smithi119 ceph-mon[29577]: osdmap e697: 8 total, 8 up, 8 in 2024-09-10T19:17:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:17:12.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:12 smithi119 ceph-mon[40674]: pgmap v1085: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T19:17:12.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-10T19:17:12.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:12 smithi119 ceph-mon[40674]: osdmap e697: 8 total, 8 up, 8 in 2024-09-10T19:17:12.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:17:13.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:12 smithi194 ceph-mon[32464]: pgmap v1085: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-09-10T19:17:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-09-10T19:17:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:12 smithi194 ceph-mon[32464]: osdmap e697: 8 total, 8 up, 8 in 2024-09-10T19:17:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-09-10T19:17:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-10T19:17:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:13 smithi119 ceph-mon[29577]: osdmap e698: 8 total, 8 up, 8 in 2024-09-10T19:17:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-10T19:17:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:13 smithi119 ceph-mon[40674]: osdmap e698: 8 total, 8 up, 8 in 2024-09-10T19:17:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:14.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-09-10T19:17:14.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:13 smithi194 ceph-mon[32464]: osdmap e698: 8 total, 8 up, 8 in 2024-09-10T19:17:14.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:14 smithi119 ceph-mon[29577]: pgmap v1088: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:17:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:17:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:14 smithi119 ceph-mon[29577]: osdmap e699: 8 total, 8 up, 8 in 2024-09-10T19:17:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142"}]: dispatch 2024-09-10T19:17:14.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:14 smithi119 ceph-mon[40674]: pgmap v1088: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:17:14.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:17:14.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:14 smithi119 ceph-mon[40674]: osdmap e699: 8 total, 8 up, 8 in 2024-09-10T19:17:14.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142"}]: dispatch 2024-09-10T19:17:15.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:14 smithi194 ceph-mon[32464]: pgmap v1088: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:17:15.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:17:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:14 smithi194 ceph-mon[32464]: osdmap e699: 8 total, 8 up, 8 in 2024-09-10T19:17:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142"}]: dispatch 2024-09-10T19:17:15.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142"}]': finished 2024-09-10T19:17:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:15 smithi119 ceph-mon[29577]: osdmap e700: 8 total, 8 up, 8 in 2024-09-10T19:17:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142"}]: dispatch 2024-09-10T19:17:15.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142"}]': finished 2024-09-10T19:17:15.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:15 smithi119 ceph-mon[40674]: osdmap e700: 8 total, 8 up, 8 in 2024-09-10T19:17:15.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:15.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142"}]: dispatch 2024-09-10T19:17:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142"}]': finished 2024-09-10T19:17:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:15 smithi194 ceph-mon[32464]: osdmap e700: 8 total, 8 up, 8 in 2024-09-10T19:17:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-142"}]: dispatch 2024-09-10T19:17:16.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:16 smithi119 ceph-mon[29577]: pgmap v1091: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:17:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:16 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:16 smithi119 ceph-mon[29577]: osdmap e701: 8 total, 8 up, 8 in 2024-09-10T19:17:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:16 smithi119 ceph-mon[40674]: pgmap v1091: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:17:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:16 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:16 smithi119 ceph-mon[40674]: osdmap e701: 8 total, 8 up, 8 in 2024-09-10T19:17:17.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:16 smithi194 ceph-mon[32464]: pgmap v1091: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:17:17.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:16 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:17.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:16 smithi194 ceph-mon[32464]: osdmap e701: 8 total, 8 up, 8 in 2024-09-10T19:17:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:17 smithi194 ceph-mon[32464]: osdmap e702: 8 total, 8 up, 8 in 2024-09-10T19:17:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:17:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:18.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:17 smithi119 ceph-mon[29577]: osdmap e702: 8 total, 8 up, 8 in 2024-09-10T19:17:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:17:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:18.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:17 smithi119 ceph-mon[40674]: osdmap e702: 8 total, 8 up, 8 in 2024-09-10T19:17:18.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:17:18.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:18.879 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:18 smithi194 ceph-mon[32464]: pgmap v1094: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:17:18.879 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:17:18.880 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:18 smithi194 ceph-mon[32464]: osdmap e703: 8 total, 8 up, 8 in 2024-09-10T19:17:18.880 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:17:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:18 smithi119 ceph-mon[29577]: pgmap v1094: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:17:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:17:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:18 smithi119 ceph-mon[29577]: osdmap e703: 8 total, 8 up, 8 in 2024-09-10T19:17:19.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:17:19.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:18 smithi119 ceph-mon[40674]: pgmap v1094: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:17:19.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:17:19.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:18 smithi119 ceph-mon[40674]: osdmap e703: 8 total, 8 up, 8 in 2024-09-10T19:17:19.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:17:19.269 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:17:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:17:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:17:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:19 smithi194 ceph-mon[32464]: osdmap e704: 8 total, 8 up, 8 in 2024-09-10T19:17:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:19 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-144"}]: dispatch 2024-09-10T19:17:20.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:17:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:19 smithi119 ceph-mon[29577]: osdmap e704: 8 total, 8 up, 8 in 2024-09-10T19:17:20.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:19 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-144"}]: dispatch 2024-09-10T19:17:20.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:17:20.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:19 smithi119 ceph-mon[40674]: osdmap e704: 8 total, 8 up, 8 in 2024-09-10T19:17:20.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:19 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-144"}]: dispatch 2024-09-10T19:17:20.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:17:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:17:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:17:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:20 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:20 smithi194 ceph-mon[32464]: pgmap v1097: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:17:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-144"}]': finished 2024-09-10T19:17:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:20 smithi194 ceph-mon[32464]: osdmap e705: 8 total, 8 up, 8 in 2024-09-10T19:17:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-144", "mode": "readproxy"}]: dispatch 2024-09-10T19:17:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:20 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:17:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:20 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-144", "mode": "readproxy"}]': finished 2024-09-10T19:17:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:20 smithi194 ceph-mon[32464]: osdmap e706: 8 total, 8 up, 8 in 2024-09-10T19:17:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[29577]: pgmap v1097: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:17:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-144"}]': finished 2024-09-10T19:17:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[29577]: osdmap e705: 8 total, 8 up, 8 in 2024-09-10T19:17:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-144", "mode": "readproxy"}]: dispatch 2024-09-10T19:17:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:17:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-144", "mode": "readproxy"}]': finished 2024-09-10T19:17:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[29577]: osdmap e706: 8 total, 8 up, 8 in 2024-09-10T19:17:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[40674]: pgmap v1097: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:17:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-144"}]': finished 2024-09-10T19:17:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[40674]: osdmap e705: 8 total, 8 up, 8 in 2024-09-10T19:17:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-144", "mode": "readproxy"}]: dispatch 2024-09-10T19:17:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:17:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-144", "mode": "readproxy"}]': finished 2024-09-10T19:17:21.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:20 smithi119 ceph-mon[40674]: osdmap e706: 8 total, 8 up, 8 in 2024-09-10T19:17:22.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:22 smithi119 ceph-mon[29577]: pgmap v1100: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:17:22.869 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:22 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:22.870 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:17:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:17:22] "GET /metrics HTTP/1.1" 200 228593 "" "Prometheus/2.43.0" 2024-09-10T19:17:22.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:22 smithi119 ceph-mon[40674]: pgmap v1100: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:17:22.870 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:22 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:23.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:22 smithi194 ceph-mon[32464]: pgmap v1100: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:17:23.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:22 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:25.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:24 smithi194 ceph-mon[32464]: pgmap v1101: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 178 B/s wr, 2 op/s 2024-09-10T19:17:25.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:24 smithi119 ceph-mon[29577]: pgmap v1101: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 178 B/s wr, 2 op/s 2024-09-10T19:17:25.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:24 smithi119 ceph-mon[40674]: pgmap v1101: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 178 B/s wr, 2 op/s 2024-09-10T19:17:27.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:26 smithi194 ceph-mon[32464]: pgmap v1102: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-09-10T19:17:27.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:26 smithi119 ceph-mon[29577]: pgmap v1102: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-09-10T19:17:27.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:26 smithi119 ceph-mon[40674]: pgmap v1102: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 913 B/s rd, 0 op/s 2024-09-10T19:17:28.880 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:28 smithi194 ceph-mon[32464]: pgmap v1103: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:17:29.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:28 smithi119 ceph-mon[29577]: pgmap v1103: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:17:29.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:28 smithi119 ceph-mon[40674]: pgmap v1103: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-09-10T19:17:29.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:17:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:17:30.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:17:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:17:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:17:31.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:30 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:31.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:30 smithi194 ceph-mon[32464]: pgmap v1104: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:17:31.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:30 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:31.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:30 smithi119 ceph-mon[29577]: pgmap v1104: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:17:31.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:30 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:31.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:30 smithi119 ceph-mon[40674]: pgmap v1104: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:17:32.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:32.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:32.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:33.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:32 smithi194 ceph-mon[32464]: pgmap v1105: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:17:33.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:17:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:32 smithi194 ceph-mon[32464]: osdmap e707: 8 total, 8 up, 8 in 2024-09-10T19:17:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144"}]: dispatch 2024-09-10T19:17:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:33.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:17:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:17:32] "GET /metrics HTTP/1.1" 200 228574 "" "Prometheus/2.43.0" 2024-09-10T19:17:33.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:32 smithi119 ceph-mon[29577]: pgmap v1105: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:17:33.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:17:33.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:32 smithi119 ceph-mon[29577]: osdmap e707: 8 total, 8 up, 8 in 2024-09-10T19:17:33.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144"}]: dispatch 2024-09-10T19:17:33.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:33.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:33.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:32 smithi119 ceph-mon[40674]: pgmap v1105: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:17:33.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:17:33.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:32 smithi119 ceph-mon[40674]: osdmap e707: 8 total, 8 up, 8 in 2024-09-10T19:17:33.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144"}]: dispatch 2024-09-10T19:17:33.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:33.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:34.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:33 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:17:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144"}]': finished 2024-09-10T19:17:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:33 smithi194 ceph-mon[32464]: osdmap e708: 8 total, 8 up, 8 in 2024-09-10T19:17:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:34.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144"}]: dispatch 2024-09-10T19:17:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:33 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:17:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144"}]': finished 2024-09-10T19:17:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:33 smithi119 ceph-mon[29577]: osdmap e708: 8 total, 8 up, 8 in 2024-09-10T19:17:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:34.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144"}]: dispatch 2024-09-10T19:17:34.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:33 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:17:34.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144"}]': finished 2024-09-10T19:17:34.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:33 smithi119 ceph-mon[40674]: osdmap e708: 8 total, 8 up, 8 in 2024-09-10T19:17:34.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:34.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-144"}]: dispatch 2024-09-10T19:17:35.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:34 smithi194 ceph-mon[32464]: pgmap v1108: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:17:35.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:34 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:34 smithi194 ceph-mon[32464]: osdmap e709: 8 total, 8 up, 8 in 2024-09-10T19:17:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:34 smithi119 ceph-mon[29577]: pgmap v1108: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:17:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:34 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:35.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:34 smithi119 ceph-mon[29577]: osdmap e709: 8 total, 8 up, 8 in 2024-09-10T19:17:35.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:34 smithi119 ceph-mon[40674]: pgmap v1108: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:17:35.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:34 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:35.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:34 smithi119 ceph-mon[40674]: osdmap e709: 8 total, 8 up, 8 in 2024-09-10T19:17:36.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:35 smithi194 ceph-mon[32464]: osdmap e710: 8 total, 8 up, 8 in 2024-09-10T19:17:36.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:17:36.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:35 smithi119 ceph-mon[29577]: osdmap e710: 8 total, 8 up, 8 in 2024-09-10T19:17:36.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:17:36.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:35 smithi119 ceph-mon[40674]: osdmap e710: 8 total, 8 up, 8 in 2024-09-10T19:17:36.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:17:37.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:36 smithi194 ceph-mon[32464]: pgmap v1111: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:17:37.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:17:37.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:36 smithi194 ceph-mon[32464]: osdmap e711: 8 total, 8 up, 8 in 2024-09-10T19:17:37.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:17:37.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:36 smithi119 ceph-mon[29577]: pgmap v1111: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:17:37.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:17:37.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:36 smithi119 ceph-mon[29577]: osdmap e711: 8 total, 8 up, 8 in 2024-09-10T19:17:37.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:17:37.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:36 smithi119 ceph-mon[40674]: pgmap v1111: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:17:37.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:17:37.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:36 smithi119 ceph-mon[40674]: osdmap e711: 8 total, 8 up, 8 in 2024-09-10T19:17:37.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:17:38.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:17:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:37 smithi194 ceph-mon[32464]: osdmap e712: 8 total, 8 up, 8 in 2024-09-10T19:17:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-146"}]: dispatch 2024-09-10T19:17:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-146"}]': finished 2024-09-10T19:17:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:37 smithi194 ceph-mon[32464]: osdmap e713: 8 total, 8 up, 8 in 2024-09-10T19:17:38.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:37 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-146", "mode": "writeback"}]: dispatch 2024-09-10T19:17:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:17:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:37 smithi119 ceph-mon[29577]: osdmap e712: 8 total, 8 up, 8 in 2024-09-10T19:17:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-146"}]: dispatch 2024-09-10T19:17:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-146"}]': finished 2024-09-10T19:17:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:37 smithi119 ceph-mon[29577]: osdmap e713: 8 total, 8 up, 8 in 2024-09-10T19:17:38.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:37 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-146", "mode": "writeback"}]: dispatch 2024-09-10T19:17:38.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:17:38.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:37 smithi119 ceph-mon[40674]: osdmap e712: 8 total, 8 up, 8 in 2024-09-10T19:17:38.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-146"}]: dispatch 2024-09-10T19:17:38.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi119-79782-111", "overlaypool": "test-rados-api-smithi119-79782-146"}]': finished 2024-09-10T19:17:38.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:37 smithi119 ceph-mon[40674]: osdmap e713: 8 total, 8 up, 8 in 2024-09-10T19:17:38.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:37 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-146", "mode": "writeback"}]: dispatch 2024-09-10T19:17:39.018 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:17:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:17:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:38 smithi194 ceph-mon[32464]: pgmap v1114: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:17:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:38 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:17:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:38 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-146", "mode": "writeback"}]': finished 2024-09-10T19:17:39.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:38 smithi194 ceph-mon[32464]: osdmap e714: 8 total, 8 up, 8 in 2024-09-10T19:17:39.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:38 smithi119 ceph-mon[29577]: pgmap v1114: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:17:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:38 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:17:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:38 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-146", "mode": "writeback"}]': finished 2024-09-10T19:17:39.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:38 smithi119 ceph-mon[29577]: osdmap e714: 8 total, 8 up, 8 in 2024-09-10T19:17:39.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:38 smithi119 ceph-mon[40674]: pgmap v1114: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:17:39.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:38 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-10T19:17:39.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:38 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi119-79782-146", "mode": "writeback"}]': finished 2024-09-10T19:17:39.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:38 smithi119 ceph-mon[40674]: osdmap e714: 8 total, 8 up, 8 in 2024-09-10T19:17:40.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:39 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:17:40.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:39 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:17:40.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:39 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_count","val": "2"}]: dispatch 2024-09-10T19:17:40.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:17:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:17:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:17:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:40 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:40 smithi194 ceph-mon[32464]: pgmap v1117: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:17:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:17:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:40 smithi194 ceph-mon[32464]: osdmap e715: 8 total, 8 up, 8 in 2024-09-10T19:17:41.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:40 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:17:41.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:40 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:41.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:40 smithi119 ceph-mon[29577]: pgmap v1117: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:17:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:17:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:40 smithi119 ceph-mon[29577]: osdmap e715: 8 total, 8 up, 8 in 2024-09-10T19:17:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:40 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:17:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:40 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:40 smithi119 ceph-mon[40674]: pgmap v1117: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:17:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_count","val": "2"}]': finished 2024-09-10T19:17:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:40 smithi119 ceph-mon[40674]: osdmap e715: 8 total, 8 up, 8 in 2024-09-10T19:17:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:40 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_period","val": "600"}]: dispatch 2024-09-10T19:17:42.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:17:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:41 smithi194 ceph-mon[32464]: osdmap e716: 8 total, 8 up, 8 in 2024-09-10T19:17:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:17:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:41 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:17:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:41 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:17:42.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:41 smithi194 ceph-mon[32464]: osdmap e717: 8 total, 8 up, 8 in 2024-09-10T19:17:42.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:17:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:41 smithi119 ceph-mon[29577]: osdmap e716: 8 total, 8 up, 8 in 2024-09-10T19:17:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:17:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:41 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:17:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:41 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:17:42.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:41 smithi119 ceph-mon[29577]: osdmap e717: 8 total, 8 up, 8 in 2024-09-10T19:17:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_period","val": "600"}]': finished 2024-09-10T19:17:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:41 smithi119 ceph-mon[40674]: osdmap e716: 8 total, 8 up, 8 in 2024-09-10T19:17:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-09-10T19:17:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:41 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-09-10T19:17:42.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:41 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "hit_set_type","val": "bloom"}]': finished 2024-09-10T19:17:42.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:41 smithi119 ceph-mon[40674]: osdmap e717: 8 total, 8 up, 8 in 2024-09-10T19:17:43.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:42 smithi194 ceph-mon[32464]: pgmap v1120: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:17:43.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:17:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:42 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:17:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:42 smithi194 ceph-mon[32464]: osdmap e718: 8 total, 8 up, 8 in 2024-09-10T19:17:43.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:42 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:17:43.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:17:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:17:42] "GET /metrics HTTP/1.1" 200 228574 "" "Prometheus/2.43.0" 2024-09-10T19:17:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:42 smithi119 ceph-mon[29577]: pgmap v1120: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:17:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:17:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:42 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:17:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:42 smithi119 ceph-mon[29577]: osdmap e718: 8 total, 8 up, 8 in 2024-09-10T19:17:43.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:42 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:17:43.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:42 smithi119 ceph-mon[40674]: pgmap v1120: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-09-10T19:17:43.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-09-10T19:17:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:42 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:17:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-09-10T19:17:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:42 smithi119 ceph-mon[40674]: osdmap e718: 8 total, 8 up, 8 in 2024-09-10T19:17:43.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:42 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "target_max_objects","val": "1"}]: dispatch 2024-09-10T19:17:45.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:44 smithi194 ceph-mon[32464]: pgmap v1123: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-10T19:17:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:44 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "target_max_objects","val": "1"}]': finished 2024-09-10T19:17:45.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:44 smithi194 ceph-mon[32464]: osdmap e719: 8 total, 8 up, 8 in 2024-09-10T19:17:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:44 smithi119 ceph-mon[29577]: pgmap v1123: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-10T19:17:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:44 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "target_max_objects","val": "1"}]': finished 2024-09-10T19:17:45.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:44 smithi119 ceph-mon[29577]: osdmap e719: 8 total, 8 up, 8 in 2024-09-10T19:17:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:44 smithi119 ceph-mon[40674]: pgmap v1123: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-10T19:17:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:44 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-146","var": "target_max_objects","val": "1"}]': finished 2024-09-10T19:17:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:44 smithi119 ceph-mon[40674]: osdmap e719: 8 total, 8 up, 8 in 2024-09-10T19:17:46.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:45 smithi194 ceph-mon[32464]: pgmap v1125: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-10T19:17:46.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:45 smithi194 ceph-mon[32464]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-10T19:17:46.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:45 smithi119 ceph-mon[29577]: pgmap v1125: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-10T19:17:46.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:45 smithi119 ceph-mon[29577]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-10T19:17:46.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:45 smithi119 ceph-mon[40674]: pgmap v1125: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-09-10T19:17:46.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:45 smithi119 ceph-mon[40674]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-09-10T19:17:48.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:48 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:48 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:48.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:48 smithi194 ceph-mon[32464]: pgmap v1126: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:17:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:48 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:48 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:48.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:48 smithi119 ceph-mon[29577]: pgmap v1126: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:17:48.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:48 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:48.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:48 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:48.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:48 smithi119 ceph-mon[40674]: pgmap v1126: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:17:49.269 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:17:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:17:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:50 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:50.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:50 smithi194 ceph-mon[32464]: pgmap v1127: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:17:50.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:17:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:17:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:17:50.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:50 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:50 smithi119 ceph-mon[29577]: pgmap v1127: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:17:50.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:50 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:50.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:50 smithi119 ceph-mon[40674]: pgmap v1127: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:17:52.707 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:52 smithi119 ceph-mon[29577]: pgmap v1128: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:17:52.707 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:17:52.707 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:52.707 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:52.707 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:52.707 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:52 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:52.707 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:52 smithi119 ceph-mon[40674]: pgmap v1128: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:17:52.707 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:17:52.708 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:52.708 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:52.708 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:52.708 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:52 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:52.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:52 smithi194 ceph-mon[32464]: pgmap v1128: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 2 op/s 2024-09-10T19:17:52.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:17:52.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:52 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:53.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:17:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:17:52] "GET /metrics HTTP/1.1" 200 228647 "" "Prometheus/2.43.0" 2024-09-10T19:17:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:53 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:17:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:53 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:17:53.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:53 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:53.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:53 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:17:53.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:53 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:17:53.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:53 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:53 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:17:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:53 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:17:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:53 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:17:54.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:54 smithi194 ceph-mon[32464]: pgmap v1129: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:17:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:54 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:54.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:54 smithi119 ceph-mon[29577]: pgmap v1129: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:17:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:54 smithi119 ceph-mon[40674]: pgmap v1129: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:17:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:54 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:55.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:17:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:55 smithi194 ceph-mon[32464]: osdmap e720: 8 total, 8 up, 8 in 2024-09-10T19:17:55.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:55 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146"}]: dispatch 2024-09-10T19:17:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:17:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:55 smithi119 ceph-mon[29577]: osdmap e720: 8 total, 8 up, 8 in 2024-09-10T19:17:55.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:55 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146"}]: dispatch 2024-09-10T19:17:55.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:17:55.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:55 smithi119 ceph-mon[40674]: osdmap e720: 8 total, 8 up, 8 in 2024-09-10T19:17:55.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:55 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146"}]: dispatch 2024-09-10T19:17:56.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:56 smithi194 ceph-mon[32464]: pgmap v1131: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:17:56.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146"}]': finished 2024-09-10T19:17:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:56 smithi194 ceph-mon[32464]: osdmap e721: 8 total, 8 up, 8 in 2024-09-10T19:17:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:56.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:56 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146"}]: dispatch 2024-09-10T19:17:56.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:56 smithi119 ceph-mon[29577]: pgmap v1131: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:17:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146"}]': finished 2024-09-10T19:17:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:56 smithi119 ceph-mon[29577]: osdmap e721: 8 total, 8 up, 8 in 2024-09-10T19:17:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:56 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146"}]: dispatch 2024-09-10T19:17:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:56 smithi119 ceph-mon[40674]: pgmap v1131: 204 pgs: 204 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-09-10T19:17:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146"}]': finished 2024-09-10T19:17:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:56 smithi119 ceph-mon[40674]: osdmap e721: 8 total, 8 up, 8 in 2024-09-10T19:17:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:17:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:56 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-146"}]: dispatch 2024-09-10T19:17:57.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:57 smithi194 ceph-mon[32464]: osdmap e722: 8 total, 8 up, 8 in 2024-09-10T19:17:57.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:57 smithi119 ceph-mon[29577]: osdmap e722: 8 total, 8 up, 8 in 2024-09-10T19:17:57.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:57 smithi119 ceph-mon[40674]: osdmap e722: 8 total, 8 up, 8 in 2024-09-10T19:17:58.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:58 smithi194 ceph-mon[32464]: pgmap v1134: 172 pgs: 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-10T19:17:58.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:58 smithi194 ceph-mon[32464]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-10T19:17:58.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:58 smithi194 ceph-mon[32464]: osdmap e723: 8 total, 8 up, 8 in 2024-09-10T19:17:58.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:58 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:17:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:58 smithi119 ceph-mon[29577]: pgmap v1134: 172 pgs: 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-10T19:17:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:58 smithi119 ceph-mon[29577]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-10T19:17:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:58 smithi119 ceph-mon[29577]: osdmap e723: 8 total, 8 up, 8 in 2024-09-10T19:17:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:58 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:17:58.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:58 smithi119 ceph-mon[40674]: pgmap v1134: 172 pgs: 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-10T19:17:58.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:58 smithi119 ceph-mon[40674]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-09-10T19:17:58.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:58 smithi119 ceph-mon[40674]: osdmap e723: 8 total, 8 up, 8 in 2024-09-10T19:17:58.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:58 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:17:59.269 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:17:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:17:59.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:59 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:17:59.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:59 smithi194 ceph-mon[32464]: osdmap e724: 8 total, 8 up, 8 in 2024-09-10T19:17:59.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:17:59 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:17:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:59 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:17:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:59 smithi119 ceph-mon[29577]: osdmap e724: 8 total, 8 up, 8 in 2024-09-10T19:17:59.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:17:59 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:17:59.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:59 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:17:59.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:59 smithi119 ceph-mon[40674]: osdmap e724: 8 total, 8 up, 8 in 2024-09-10T19:17:59.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:17:59 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-09-10T19:18:00.424 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:18:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:18:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:18:00.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:00 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:00 smithi194 ceph-mon[32464]: pgmap v1137: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-10T19:18:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:18:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:00 smithi194 ceph-mon[32464]: osdmap e725: 8 total, 8 up, 8 in 2024-09-10T19:18:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:00 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148"}]: dispatch 2024-09-10T19:18:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:00 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:00 smithi119 ceph-mon[29577]: pgmap v1137: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-10T19:18:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:18:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:00 smithi119 ceph-mon[29577]: osdmap e725: 8 total, 8 up, 8 in 2024-09-10T19:18:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:00 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148"}]: dispatch 2024-09-10T19:18:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:00 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:00 smithi119 ceph-mon[40674]: pgmap v1137: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-10T19:18:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148", "force_nonempty": "--force-nonempty" }]': finished 2024-09-10T19:18:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:00 smithi119 ceph-mon[40674]: osdmap e725: 8 total, 8 up, 8 in 2024-09-10T19:18:00.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:00 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148"}]: dispatch 2024-09-10T19:18:01.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148"}]': finished 2024-09-10T19:18:01.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:01 smithi194 ceph-mon[32464]: osdmap e726: 8 total, 8 up, 8 in 2024-09-10T19:18:01.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:01.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:01 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148"}]: dispatch 2024-09-10T19:18:01.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148"}]': finished 2024-09-10T19:18:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:01 smithi119 ceph-mon[29577]: osdmap e726: 8 total, 8 up, 8 in 2024-09-10T19:18:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:01.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:01 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148"}]: dispatch 2024-09-10T19:18:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148"}]': finished 2024-09-10T19:18:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:01 smithi119 ceph-mon[40674]: osdmap e726: 8 total, 8 up, 8 in 2024-09-10T19:18:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:01.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:01 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-148"}]: dispatch 2024-09-10T19:18:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:02 smithi119 ceph-mon[29577]: pgmap v1140: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:02.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:02 smithi119 ceph-mon[29577]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:18:02.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:02 smithi119 ceph-mon[29577]: osdmap e727: 8 total, 8 up, 8 in 2024-09-10T19:18:02.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:18:02.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:02 smithi119 ceph-mon[40674]: pgmap v1140: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:02 smithi119 ceph-mon[40674]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:18:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:02 smithi119 ceph-mon[40674]: osdmap e727: 8 total, 8 up, 8 in 2024-09-10T19:18:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:18:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:02.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:02 smithi194 ceph-mon[32464]: pgmap v1140: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:02 smithi194 ceph-mon[32464]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:18:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:02 smithi194 ceph-mon[32464]: osdmap e727: 8 total, 8 up, 8 in 2024-09-10T19:18:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:18:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:03.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:18:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:18:02] "GET /metrics HTTP/1.1" 200 226133 "" "Prometheus/2.43.0" 2024-09-10T19:18:03.768 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:18:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a[65800]: logger=sqlstore.transactions t=2024-09-10T19:18:03.334453525Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T19:18:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:03 smithi194 ceph-mon[32464]: osdmap e728: 8 total, 8 up, 8 in 2024-09-10T19:18:03.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:03 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:18:03.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:03 smithi119 ceph-mon[29577]: osdmap e728: 8 total, 8 up, 8 in 2024-09-10T19:18:03.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:03 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:18:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:03 smithi119 ceph-mon[40674]: osdmap e728: 8 total, 8 up, 8 in 2024-09-10T19:18:03.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:03 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:18:04.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:04 smithi194 ceph-mon[32464]: pgmap v1143: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:18:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:18:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:04 smithi194 ceph-mon[32464]: osdmap e729: 8 total, 8 up, 8 in 2024-09-10T19:18:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:04.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:04 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-150"}]: dispatch 2024-09-10T19:18:04.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:04 smithi119 ceph-mon[29577]: pgmap v1143: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:18:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:18:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:04 smithi119 ceph-mon[29577]: osdmap e729: 8 total, 8 up, 8 in 2024-09-10T19:18:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:04 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-150"}]: dispatch 2024-09-10T19:18:04.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:04 smithi119 ceph-mon[40674]: pgmap v1143: 204 pgs: 32 creating+peering, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:18:04.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:18:04.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:04 smithi119 ceph-mon[40674]: osdmap e729: 8 total, 8 up, 8 in 2024-09-10T19:18:04.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:04.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:04 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-150"}]: dispatch 2024-09-10T19:18:05.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:05 smithi194 ceph-mon[32464]: osdmap e730: 8 total, 8 up, 8 in 2024-09-10T19:18:05.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:05 smithi119 ceph-mon[29577]: osdmap e730: 8 total, 8 up, 8 in 2024-09-10T19:18:05.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:05 smithi119 ceph-mon[40674]: osdmap e730: 8 total, 8 up, 8 in 2024-09-10T19:18:06.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:06 smithi194 ceph-mon[32464]: pgmap v1146: 172 pgs: 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:18:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:06 smithi194 ceph-mon[32464]: osdmap e731: 8 total, 8 up, 8 in 2024-09-10T19:18:06.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:06 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:18:06.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:06 smithi119 ceph-mon[29577]: pgmap v1146: 172 pgs: 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:18:06.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:06 smithi119 ceph-mon[29577]: osdmap e731: 8 total, 8 up, 8 in 2024-09-10T19:18:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:06 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:18:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:06 smithi119 ceph-mon[40674]: pgmap v1146: 172 pgs: 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-09-10T19:18:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:06 smithi119 ceph-mon[40674]: osdmap e731: 8 total, 8 up, 8 in 2024-09-10T19:18:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:06 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:18:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:18:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:07 smithi194 ceph-mon[32464]: osdmap e732: 8 total, 8 up, 8 in 2024-09-10T19:18:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-152"}]: dispatch 2024-09-10T19:18:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:07 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:18:07.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:18:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:07 smithi119 ceph-mon[29577]: osdmap e732: 8 total, 8 up, 8 in 2024-09-10T19:18:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-152"}]: dispatch 2024-09-10T19:18:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:07 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:18:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:18:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:07 smithi119 ceph-mon[40674]: osdmap e732: 8 total, 8 up, 8 in 2024-09-10T19:18:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-152"}]: dispatch 2024-09-10T19:18:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:07 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:18:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:08 smithi194 ceph-mon[32464]: pgmap v1149: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:18:08.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:08 smithi194 ceph-mon[32464]: osdmap e733: 8 total, 8 up, 8 in 2024-09-10T19:18:08.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:08 smithi119 ceph-mon[29577]: pgmap v1149: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:18:08.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:08 smithi119 ceph-mon[29577]: osdmap e733: 8 total, 8 up, 8 in 2024-09-10T19:18:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:08 smithi119 ceph-mon[40674]: pgmap v1149: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:18:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:08 smithi119 ceph-mon[40674]: osdmap e733: 8 total, 8 up, 8 in 2024-09-10T19:18:09.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:18:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:18:09.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:09 smithi194 ceph-mon[32464]: osdmap e734: 8 total, 8 up, 8 in 2024-09-10T19:18:09.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:18:09.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:09 smithi119 ceph-mon[29577]: osdmap e734: 8 total, 8 up, 8 in 2024-09-10T19:18:09.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:18:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:09 smithi119 ceph-mon[40674]: osdmap e734: 8 total, 8 up, 8 in 2024-09-10T19:18:09.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:18:10.503 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:18:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:18:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:18:10.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:10 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:10 smithi194 ceph-mon[32464]: pgmap v1152: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-10T19:18:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:18:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:10 smithi194 ceph-mon[32464]: osdmap e735: 8 total, 8 up, 8 in 2024-09-10T19:18:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-111","var": "dedup_tier","val": "test-rados-api-smithi119-79782-154"}]: dispatch 2024-09-10T19:18:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:10 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-154"}]: dispatch 2024-09-10T19:18:10.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:10 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:10 smithi119 ceph-mon[29577]: pgmap v1152: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-10T19:18:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:18:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:10 smithi119 ceph-mon[29577]: osdmap e735: 8 total, 8 up, 8 in 2024-09-10T19:18:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-111","var": "dedup_tier","val": "test-rados-api-smithi119-79782-154"}]: dispatch 2024-09-10T19:18:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:10 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-154"}]: dispatch 2024-09-10T19:18:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:10 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:10 smithi119 ceph-mon[40674]: pgmap v1152: 204 pgs: 32 unknown, 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-10T19:18:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi119-79782-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T19:18:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:10 smithi119 ceph-mon[40674]: osdmap e735: 8 total, 8 up, 8 in 2024-09-10T19:18:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi119-79782-111","var": "dedup_tier","val": "test-rados-api-smithi119-79782-154"}]: dispatch 2024-09-10T19:18:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:10.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:10 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi119-79782-111", "tierpool": "test-rados-api-smithi119-79782-154"}]: dispatch 2024-09-10T19:18:11.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:11 smithi194 ceph-mon[32464]: osdmap e736: 8 total, 8 up, 8 in 2024-09-10T19:18:11.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:11 smithi194 ceph-mon[32464]: osdmap e737: 8 total, 8 up, 8 in 2024-09-10T19:18:11.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:11 smithi119 ceph-mon[29577]: osdmap e736: 8 total, 8 up, 8 in 2024-09-10T19:18:11.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:11 smithi119 ceph-mon[29577]: osdmap e737: 8 total, 8 up, 8 in 2024-09-10T19:18:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:11 smithi119 ceph-mon[40674]: osdmap e736: 8 total, 8 up, 8 in 2024-09-10T19:18:11.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:11 smithi119 ceph-mon[40674]: osdmap e737: 8 total, 8 up, 8 in 2024-09-10T19:18:12.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:12 smithi194 ceph-mon[32464]: pgmap v1155: 172 pgs: 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-10T19:18:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:12 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:12.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:12 smithi119 ceph-mon[29577]: pgmap v1155: 172 pgs: 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-10T19:18:12.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:12 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:12.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:18:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:18:12] "GET /metrics HTTP/1.1" 200 226133 "" "Prometheus/2.43.0" 2024-09-10T19:18:12.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:12 smithi119 ceph-mon[40674]: pgmap v1155: 172 pgs: 172 active+clean; 4.4 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 2.2 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-10T19:18:12.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:12 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:13.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7902 ms) 2024-09-10T19:18:13.512 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-10T19:18:13.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13060 ms) 2024-09-10T19:18:13.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-10T19:18:13.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8046 ms) 2024-09-10T19:18:13.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-10T19:18:13.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13078 ms) 2024-09-10T19:18:13.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-10T19:18:13.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8840 ms) 2024-09-10T19:18:13.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-10T19:18:13.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8051 ms) 2024-09-10T19:18:13.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-10T19:18:13.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-10T19:18:13.513 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: ok, hit_set contains 255:602f83fe:::foo:head 2024-09-10T19:18:13.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9053 ms) 2024-09-10T19:18:13.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-10T19:18:13.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995808,0 2024-09-10T19:18:13.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: first is 1725995808 2024-09-10T19:18:13.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995808,0 2024-09-10T19:18:13.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995808,0 2024-09-10T19:18:13.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995808,1725995810,1725995811,0 2024-09-10T19:18:13.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995808,1725995810,1725995811,0 2024-09-10T19:18:13.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995808,1725995810,1725995811,0 2024-09-10T19:18:13.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995808,1725995810,1725995811,1725995813,1725995814,0 2024-09-10T19:18:13.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995808,1725995810,1725995811,1725995813,1725995814,0 2024-09-10T19:18:13.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995808,1725995810,1725995811,1725995813,1725995814,0 2024-09-10T19:18:13.514 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995808,1725995810,1725995811,1725995813,1725995814,1725995816,1725995817,0 2024-09-10T19:18:13.515 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995808,1725995810,1725995811,1725995813,1725995814,1725995816,1725995817,0 2024-09-10T19:18:13.515 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995808,1725995810,1725995811,1725995813,1725995814,1725995816,1725995817,0 2024-09-10T19:18:13.515 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: got ls 1725995811,1725995813,1725995814,1725995816,1725995817,1725995819,1725995820,0 2024-09-10T19:18:13.515 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: first now 1725995811, trimmed 2024-09-10T19:18:13.515 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21066 ms) 2024-09-10T19:18:13.515 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-10T19:18:13.515 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: foo0 2024-09-10T19:18:13.515 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-10T19:18:13.515 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14069 ms) 2024-09-10T19:18:13.515 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-10T19:18:13.515 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18077 ms) 2024-09-10T19:18:13.515 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-10T19:18:13.515 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22748 ms) 2024-09-10T19:18:13.516 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-10T19:18:13.516 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5041 ms) 2024-09-10T19:18:13.516 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-10T19:18:13.516 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3022 ms) 2024-09-10T19:18:13.516 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-10T19:18:13.516 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3031 ms) 2024-09-10T19:18:13.516 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-10T19:18:13.516 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3012 ms) 2024-09-10T19:18:13.516 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (233712 ms total) 2024-09-10T19:18:13.516 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: 2024-09-10T19:18:13.516 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-10T19:18:13.516 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (852146 ms total) 2024-09-10T19:18:13.517 INFO:tasks.workunit.client.0.smithi119.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-10T19:18:13.520 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.520 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79453 2024-09-10T19:18:13.520 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79453 2024-09-10T19:18:13.520 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.520 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79666 2024-09-10T19:18:13.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79666 2024-09-10T19:18:13.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=80185 2024-09-10T19:18:13.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 80185 2024-09-10T19:18:13.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79929 2024-09-10T19:18:13.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79929 2024-09-10T19:18:13.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79753 2024-09-10T19:18:13.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79753 2024-09-10T19:18:13.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79541 2024-09-10T19:18:13.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79541 2024-09-10T19:18:13.521 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79849 2024-09-10T19:18:13.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79849 2024-09-10T19:18:13.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79567 2024-09-10T19:18:13.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79567 2024-09-10T19:18:13.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=80087 2024-09-10T19:18:13.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 80087 2024-09-10T19:18:13.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79426 2024-09-10T19:18:13.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79426 2024-09-10T19:18:13.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79490 2024-09-10T19:18:13.522 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79490 2024-09-10T19:18:13.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=80124 2024-09-10T19:18:13.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 80124 2024-09-10T19:18:13.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79417 2024-09-10T19:18:13.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79417 2024-09-10T19:18:13.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79437 2024-09-10T19:18:13.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79437 2024-09-10T19:18:13.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79701 2024-09-10T19:18:13.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79701 2024-09-10T19:18:13.523 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79827 2024-09-10T19:18:13.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79827 2024-09-10T19:18:13.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79721 2024-09-10T19:18:13.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79721 2024-09-10T19:18:13.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=80155 2024-09-10T19:18:13.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 80155 2024-09-10T19:18:13.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=80072 2024-09-10T19:18:13.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 80072 2024-09-10T19:18:13.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79617 2024-09-10T19:18:13.524 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79617 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79996 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79996 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79410 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79410 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79470 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79470 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79892 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79892 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=79510 2024-09-10T19:18:13.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 79510 2024-09-10T19:18:13.526 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.526 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=80039 2024-09-10T19:18:13.526 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 80039 2024-09-10T19:18:13.526 INFO:tasks.workunit.client.0.smithi119.stderr:+ for t in "${!pids[@]}" 2024-09-10T19:18:13.526 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=80060 2024-09-10T19:18:13.526 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 80060 2024-09-10T19:18:13.526 INFO:tasks.workunit.client.0.smithi119.stderr:+ exit 0 2024-09-10T19:18:13.526 INFO:tasks.workunit.client.0.smithi119.stderr:+ cleanup 2024-09-10T19:18:13.526 INFO:tasks.workunit.client.0.smithi119.stderr:+ pkill -P 79405 2024-09-10T19:18:13.533 INFO:tasks.workunit.client.0.smithi119.stderr:+ true 2024-09-10T19:18:13.533 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T19:18:13.533 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T19:18:13.564 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2024-09-10T19:18:13.565 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2024-09-10T19:18:13.565 DEBUG:teuthology.orchestra.run.smithi119:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=fad5356a94e6044ce1062d748daa5a0f2ed53a7d TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-09-10T19:18:13.624 INFO:tasks.workunit.client.0.smithi119.stderr:++ uuidgen 2024-09-10T19:18:13.634 INFO:tasks.workunit.client.0.smithi119.stderr:+ p=791ac3f1-f414-442d-8950-49875e1f19bf 2024-09-10T19:18:13.635 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd pool create 791ac3f1-f414-442d-8950-49875e1f19bf 12 2024-09-10T19:18:13.715 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.711+0000 7ffa6889a640 1 -- 172.21.15.119:0/3544504998 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa640fbd30 msgr2=0x7ffa641083d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:13.715 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.711+0000 7ffa6889a640 1 --2- 172.21.15.119:0/3544504998 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa640fbd30 0x7ffa641083d0 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c009cd0 tx=0x7ffa5c004d10 comp rx=0 tx=0).stop 2024-09-10T19:18:13.715 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.711+0000 7ffa6889a640 1 -- 172.21.15.119:0/3544504998 shutdown_connections 2024-09-10T19:18:13.715 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.711+0000 7ffa6889a640 1 --2- 172.21.15.119:0/3544504998 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa640fbd30 0x7ffa641083d0 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:13.715 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.711+0000 7ffa6889a640 1 --2- 172.21.15.119:0/3544504998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa64101f90 0x7ffa640fb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:13.715 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.711+0000 7ffa6889a640 1 --2- 172.21.15.119:0/3544504998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa641015f0 0x7ffa641019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:13.715 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.711+0000 7ffa6889a640 1 -- 172.21.15.119:0/3544504998 >> 172.21.15.119:0/3544504998 conn(0x7ffa64072510 msgr2=0x7ffa640f9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:18:13.715 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.711+0000 7ffa6889a640 1 -- 172.21.15.119:0/3544504998 shutdown_connections 2024-09-10T19:18:13.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.711+0000 7ffa6889a640 1 -- 172.21.15.119:0/3544504998 wait complete. 2024-09-10T19:18:13.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6889a640 1 Processor -- start 2024-09-10T19:18:13.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6889a640 1 -- start start 2024-09-10T19:18:13.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6889a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa640fbd30 0x7ffa640fd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:13.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6889a640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa641015f0 0x7ffa640fd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:13.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6889a640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa64101f90 0x7ffa640fdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:13.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6889a640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6411b060 con 0x7ffa64101f90 2024-09-10T19:18:13.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6889a640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6411b1d0 con 0x7ffa641015f0 2024-09-10T19:18:13.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6889a640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6411b390 con 0x7ffa640fbd30 2024-09-10T19:18:13.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6259c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa640fbd30 0x7ffa640fd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:13.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6259c640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa640fbd30 0x7ffa640fd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:60010/0 (socket says 172.21.15.119:60010) 2024-09-10T19:18:13.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6259c640 1 -- 172.21.15.119:0/1603007101 learned_addr learned my addr 172.21.15.119:0/1603007101 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:18:13.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa62d9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa64101f90 0x7ffa640fdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:13.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa61d9b640 1 --2- 172.21.15.119:0/1603007101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa641015f0 0x7ffa640fd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:13.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6259c640 1 -- 172.21.15.119:0/1603007101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa641015f0 msgr2=0x7ffa640fd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:13.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6259c640 1 --2- 172.21.15.119:0/1603007101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa641015f0 0x7ffa640fd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:13.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6259c640 1 -- 172.21.15.119:0/1603007101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa64101f90 msgr2=0x7ffa640fdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:13.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6259c640 1 --2- 172.21.15.119:0/1603007101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa64101f90 0x7ffa640fdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:13.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6259c640 1 -- 172.21.15.119:0/1603007101 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa54005040 con 0x7ffa640fbd30 2024-09-10T19:18:13.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa61d9b640 1 --2- 172.21.15.119:0/1603007101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa641015f0 0x7ffa640fd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T19:18:13.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa62d9d640 1 --2- 172.21.15.119:0/1603007101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa64101f90 0x7ffa640fdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:18:13.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.712+0000 7ffa6259c640 1 --2- 172.21.15.119:0/1603007101 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa640fbd30 0x7ffa640fd230 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7ffa54009780 tx=0x7ffa54009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:18:13.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.713+0000 7ffa52ffd640 1 -- 172.21.15.119:0/1603007101 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa540099c0 con 0x7ffa640fbd30 2024-09-10T19:18:13.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.713+0000 7ffa52ffd640 1 -- 172.21.15.119:0/1603007101 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7ffa5400f040 con 0x7ffa640fbd30 2024-09-10T19:18:13.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.713+0000 7ffa52ffd640 1 -- 172.21.15.119:0/1603007101 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa54013ca0 con 0x7ffa640fbd30 2024-09-10T19:18:13.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.713+0000 7ffa6889a640 1 -- 172.21.15.119:0/1603007101 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa5c0098b0 con 0x7ffa640fbd30 2024-09-10T19:18:13.718 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.713+0000 7ffa6889a640 1 -- 172.21.15.119:0/1603007101 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6411b950 con 0x7ffa640fbd30 2024-09-10T19:18:13.718 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.714+0000 7ffa6889a640 1 -- 172.21.15.119:0/1603007101 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa640fcd70 con 0x7ffa640fbd30 2024-09-10T19:18:13.718 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.714+0000 7ffa52ffd640 1 -- 172.21.15.119:0/1603007101 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7ffa5400ebb0 con 0x7ffa640fbd30 2024-09-10T19:18:13.719 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.715+0000 7ffa52ffd640 1 --2- 172.21.15.119:0/1603007101 >> v2:172.21.15.119:6800/3118961733 conn(0x7ffa4c06ef80 0x7ffa4c071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:13.719 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.715+0000 7ffa61d9b640 1 --2- 172.21.15.119:0/1603007101 >> v2:172.21.15.119:6800/3118961733 conn(0x7ffa4c06ef80 0x7ffa4c071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:13.719 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.715+0000 7ffa52ffd640 1 -- 172.21.15.119:0/1603007101 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(739..739 src has 1..739) v4 ==== 7467+0+0 (secure 0 0 0) 0x7ffa5408e8b0 con 0x7ffa640fbd30 2024-09-10T19:18:13.719 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.715+0000 7ffa61d9b640 1 --2- 172.21.15.119:0/1603007101 >> v2:172.21.15.119:6800/3118961733 conn(0x7ffa4c06ef80 0x7ffa4c071430 secure :-1 s=READY pgs=3705 cs=0 l=1 rev1=1 crypto rx=0x7ffa58006fd0 tx=0x7ffa58008040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:18:13.721 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.719+0000 7ffa52ffd640 1 -- 172.21.15.119:0/1603007101 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7ffa5405b280 con 0x7ffa640fbd30 2024-09-10T19:18:13.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:18:13.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:13 smithi194 ceph-mon[32464]: osdmap e738: 8 total, 8 up, 8 in 2024-09-10T19:18:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:13 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:18:13.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:13 smithi194 ceph-mon[32464]: osdmap e739: 8 total, 8 up, 8 in 2024-09-10T19:18:13.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:18:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:13 smithi119 ceph-mon[29577]: osdmap e738: 8 total, 8 up, 8 in 2024-09-10T19:18:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:13 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:18:13.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:13 smithi119 ceph-mon[29577]: osdmap e739: 8 total, 8 up, 8 in 2024-09-10T19:18:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:18:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:13 smithi119 ceph-mon[40674]: osdmap e738: 8 total, 8 up, 8 in 2024-09-10T19:18:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-111"}]: dispatch 2024-09-10T19:18:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:13 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2673652446' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi119-79782-111"}]': finished 2024-09-10T19:18:13.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:13 smithi119 ceph-mon[40674]: osdmap e739: 8 total, 8 up, 8 in 2024-09-10T19:18:13.880 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:13.878+0000 7ffa6889a640 1 -- 172.21.15.119:0/1603007101 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12} v 0) v1 -- 0x7ffa6410aac0 con 0x7ffa640fbd30 2024-09-10T19:18:14.525 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.523+0000 7ffa52ffd640 1 -- 172.21.15.119:0/1603007101 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]=0 pool '791ac3f1-f414-442d-8950-49875e1f19bf' created v740) v1 ==== 176+0+0 (secure 0 0 0) 0x7ffa54013e00 con 0x7ffa640fbd30 2024-09-10T19:18:14.623 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.621+0000 7ffa6889a640 1 -- 172.21.15.119:0/1603007101 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12} v 0) v1 -- 0x7ffa6404bf60 con 0x7ffa640fbd30 2024-09-10T19:18:14.624 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.622+0000 7ffa52ffd640 1 -- 172.21.15.119:0/1603007101 <== mon.2 v2:172.21.15.119:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]=0 pool '791ac3f1-f414-442d-8950-49875e1f19bf' already exists v740) v1 ==== 183+0+0 (secure 0 0 0) 0x7ffa5405ac20 con 0x7ffa640fbd30 2024-09-10T19:18:14.624 INFO:tasks.workunit.client.0.smithi119.stderr:pool '791ac3f1-f414-442d-8950-49875e1f19bf' already exists 2024-09-10T19:18:14.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.624+0000 7ffa6889a640 1 -- 172.21.15.119:0/1603007101 >> v2:172.21.15.119:6800/3118961733 conn(0x7ffa4c06ef80 msgr2=0x7ffa4c071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:14.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.624+0000 7ffa6889a640 1 --2- 172.21.15.119:0/1603007101 >> v2:172.21.15.119:6800/3118961733 conn(0x7ffa4c06ef80 0x7ffa4c071430 secure :-1 s=READY pgs=3705 cs=0 l=1 rev1=1 crypto rx=0x7ffa58006fd0 tx=0x7ffa58008040 comp rx=0 tx=0).stop 2024-09-10T19:18:14.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.624+0000 7ffa6889a640 1 -- 172.21.15.119:0/1603007101 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa640fbd30 msgr2=0x7ffa640fd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:14.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.624+0000 7ffa6889a640 1 --2- 172.21.15.119:0/1603007101 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa640fbd30 0x7ffa640fd230 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7ffa54009780 tx=0x7ffa54009da0 comp rx=0 tx=0).stop 2024-09-10T19:18:14.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.624+0000 7ffa6889a640 1 -- 172.21.15.119:0/1603007101 shutdown_connections 2024-09-10T19:18:14.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.624+0000 7ffa6889a640 1 --2- 172.21.15.119:0/1603007101 >> v2:172.21.15.119:6800/3118961733 conn(0x7ffa4c06ef80 0x7ffa4c071430 unknown :-1 s=CLOSED pgs=3705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:14.627 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.624+0000 7ffa6889a640 1 --2- 172.21.15.119:0/1603007101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa64101f90 0x7ffa640fdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:14.627 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.624+0000 7ffa6889a640 1 --2- 172.21.15.119:0/1603007101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa641015f0 0x7ffa640fd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:14.627 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.624+0000 7ffa6889a640 1 --2- 172.21.15.119:0/1603007101 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ffa640fbd30 0x7ffa640fd230 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:14.627 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.624+0000 7ffa6889a640 1 -- 172.21.15.119:0/1603007101 >> 172.21.15.119:0/1603007101 conn(0x7ffa64072510 msgr2=0x7ffa640fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:18:14.627 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.624+0000 7ffa6889a640 1 -- 172.21.15.119:0/1603007101 shutdown_connections 2024-09-10T19:18:14.627 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.624+0000 7ffa6889a640 1 -- 172.21.15.119:0/1603007101 wait complete. 2024-09-10T19:18:14.634 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd pool set-quota 791ac3f1-f414-442d-8950-49875e1f19bf max_objects 10 2024-09-10T19:18:14.712 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.711+0000 7f05deab8640 1 -- 172.21.15.119:0/826524135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05d80fc720 msgr2=0x7f05d80fcb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:14.713 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.711+0000 7f05deab8640 1 --2- 172.21.15.119:0/826524135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05d80fc720 0x7f05d80fcb90 secure :-1 s=READY pgs=2428 cs=0 l=1 rev1=1 crypto rx=0x7f05c4004980 tx=0x7f05c4009da0 comp rx=0 tx=0).stop 2024-09-10T19:18:14.713 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.711+0000 7f05deab8640 1 -- 172.21.15.119:0/826524135 shutdown_connections 2024-09-10T19:18:14.713 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.711+0000 7f05deab8640 1 --2- 172.21.15.119:0/826524135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f05d80fd0d0 0x7f05d81094f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:14.713 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.711+0000 7f05deab8640 1 --2- 172.21.15.119:0/826524135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05d80fc720 0x7f05d80fcb90 unknown :-1 s=CLOSED pgs=2428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:14.713 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.711+0000 7f05deab8640 1 --2- 172.21.15.119:0/826524135 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f05d8102740 0x7f05d8102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:14.713 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.711+0000 7f05deab8640 1 -- 172.21.15.119:0/826524135 >> 172.21.15.119:0/826524135 conn(0x7f05d80f82c0 msgr2=0x7f05d80fa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:18:14.713 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.711+0000 7f05deab8640 1 -- 172.21.15.119:0/826524135 shutdown_connections 2024-09-10T19:18:14.713 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.711+0000 7f05deab8640 1 -- 172.21.15.119:0/826524135 wait complete. 2024-09-10T19:18:14.715 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.711+0000 7f05deab8640 1 Processor -- start 2024-09-10T19:18:14.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.711+0000 7f05deab8640 1 -- start start 2024-09-10T19:18:14.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05deab8640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f05d80fc720 0x7f05d8112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:14.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05deab8640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05d80fd0d0 0x7f05d8113120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:14.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05deab8640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f05d8102740 0x7f05d810cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:14.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05deab8640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d810d3e0 con 0x7f05d80fd0d0 2024-09-10T19:18:14.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05deab8640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05d810d550 con 0x7f05d8102740 2024-09-10T19:18:14.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05deab8640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05d810d710 con 0x7f05d80fc720 2024-09-10T19:18:14.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05dc854640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f05d80fc720 0x7f05d8112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:14.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05d7fff640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05d80fd0d0 0x7f05d8113120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:14.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05dc854640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f05d80fc720 0x7f05d8112be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:60028/0 (socket says 172.21.15.119:60028) 2024-09-10T19:18:14.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05dc854640 1 -- 172.21.15.119:0/2232113587 learned_addr learned my addr 172.21.15.119:0/2232113587 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:18:14.716 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05dd055640 1 --2- 172.21.15.119:0/2232113587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f05d8102740 0x7f05d810cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:14.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05dc854640 1 -- 172.21.15.119:0/2232113587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f05d8102740 msgr2=0x7f05d810cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:14.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05dc854640 1 --2- 172.21.15.119:0/2232113587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f05d8102740 0x7f05d810cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:14.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05dc854640 1 -- 172.21.15.119:0/2232113587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05d80fd0d0 msgr2=0x7f05d8113120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:14.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05dc854640 1 --2- 172.21.15.119:0/2232113587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05d80fd0d0 0x7f05d8113120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:14.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05dc854640 1 -- 172.21.15.119:0/2232113587 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05cc005040 con 0x7f05d80fc720 2024-09-10T19:18:14.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.712+0000 7f05d7fff640 1 --2- 172.21.15.119:0/2232113587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05d80fd0d0 0x7f05d8113120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:18:14.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.713+0000 7f05dc854640 1 --2- 172.21.15.119:0/2232113587 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f05d80fc720 0x7f05d8112be0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009780 tx=0x7f05cc009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:18:14.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.713+0000 7f05d57fa640 1 -- 172.21.15.119:0/2232113587 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc0099c0 con 0x7f05d80fc720 2024-09-10T19:18:14.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.713+0000 7f05d57fa640 1 -- 172.21.15.119:0/2232113587 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f05cc00f040 con 0x7f05d80fc720 2024-09-10T19:18:14.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.713+0000 7f05deab8640 1 -- 172.21.15.119:0/2232113587 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c4005040 con 0x7f05d80fc720 2024-09-10T19:18:14.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.713+0000 7f05d57fa640 1 -- 172.21.15.119:0/2232113587 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05cc013ca0 con 0x7f05d80fc720 2024-09-10T19:18:14.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.713+0000 7f05deab8640 1 -- 172.21.15.119:0/2232113587 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05d8119a50 con 0x7f05d80fc720 2024-09-10T19:18:14.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.714+0000 7f05deab8640 1 -- 172.21.15.119:0/2232113587 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05d80fde60 con 0x7f05d80fc720 2024-09-10T19:18:14.717 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.714+0000 7f05d57fa640 1 -- 172.21.15.119:0/2232113587 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f05cc00ebb0 con 0x7f05d80fc720 2024-09-10T19:18:14.718 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.715+0000 7f05d57fa640 1 --2- 172.21.15.119:0/2232113587 >> v2:172.21.15.119:6800/3118961733 conn(0x7f05c806ef80 0x7f05c8071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:14.718 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.715+0000 7f05d7fff640 1 --2- 172.21.15.119:0/2232113587 >> v2:172.21.15.119:6800/3118961733 conn(0x7f05c806ef80 0x7f05c8071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:14.718 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.715+0000 7f05d57fa640 1 -- 172.21.15.119:0/2232113587 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(740..740 src has 1..740) v4 ==== 7842+0+0 (secure 0 0 0) 0x7f05cc08eaf0 con 0x7f05d80fc720 2024-09-10T19:18:14.718 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.715+0000 7f05d7fff640 1 --2- 172.21.15.119:0/2232113587 >> v2:172.21.15.119:6800/3118961733 conn(0x7f05c806ef80 0x7f05c8071430 secure :-1 s=READY pgs=3706 cs=0 l=1 rev1=1 crypto rx=0x7f05d810e4f0 tx=0x7f05c4009da0 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:18:14.720 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.719+0000 7f05d57fa640 1 -- 172.21.15.119:0/2232113587 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f05cc05b2c0 con 0x7f05d80fc720 2024-09-10T19:18:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:14 smithi119 ceph-mon[29577]: pgmap v1158: 164 pgs: 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:18:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:14 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:18:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:14 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1603007101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]: dispatch 2024-09-10T19:18:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:14 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]: dispatch 2024-09-10T19:18:14.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:14 smithi119 ceph-mon[40674]: pgmap v1158: 164 pgs: 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:18:14.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:14 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:18:14.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:14 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1603007101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]: dispatch 2024-09-10T19:18:14.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:14 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]: dispatch 2024-09-10T19:18:14.880 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:14.878+0000 7f05deab8640 1 -- 172.21.15.119:0/2232113587 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f05d810bba0 con 0x7f05d80fc720 2024-09-10T19:18:15.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:14 smithi194 ceph-mon[32464]: pgmap v1158: 164 pgs: 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:18:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:14 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:18:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:14 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1603007101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]: dispatch 2024-09-10T19:18:15.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:14 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]: dispatch 2024-09-10T19:18:15.525 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:15.524+0000 7f05d57fa640 1 -- 172.21.15.119:0/2232113587 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 791ac3f1-f414-442d-8950-49875e1f19bf v741) v1 ==== 223+0+0 (secure 0 0 0) 0x7f05cc013e00 con 0x7f05d80fc720 2024-09-10T19:18:15.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:15.623+0000 7f05deab8640 1 -- 172.21.15.119:0/2232113587 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f05d810e2a0 con 0x7f05d80fc720 2024-09-10T19:18:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]': finished 2024-09-10T19:18:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[29577]: osdmap e740: 8 total, 8 up, 8 in 2024-09-10T19:18:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1603007101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]: dispatch 2024-09-10T19:18:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]: dispatch 2024-09-10T19:18:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2232113587' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:18:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:18:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]': finished 2024-09-10T19:18:15.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[29577]: osdmap e741: 8 total, 8 up, 8 in 2024-09-10T19:18:15.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]': finished 2024-09-10T19:18:15.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[40674]: osdmap e740: 8 total, 8 up, 8 in 2024-09-10T19:18:15.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1603007101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]: dispatch 2024-09-10T19:18:15.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]: dispatch 2024-09-10T19:18:15.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2232113587' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:18:15.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:18:15.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]': finished 2024-09-10T19:18:15.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:15 smithi119 ceph-mon[40674]: osdmap e741: 8 total, 8 up, 8 in 2024-09-10T19:18:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]': finished 2024-09-10T19:18:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:15 smithi194 ceph-mon[32464]: osdmap e740: 8 total, 8 up, 8 in 2024-09-10T19:18:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1603007101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]: dispatch 2024-09-10T19:18:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pg_num": 12}]: dispatch 2024-09-10T19:18:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:15 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2232113587' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:18:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:18:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:15 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]': finished 2024-09-10T19:18:16.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:15 smithi194 ceph-mon[32464]: osdmap e741: 8 total, 8 up, 8 in 2024-09-10T19:18:16.526 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.524+0000 7f05d57fa640 1 -- 172.21.15.119:0/2232113587 <== mon.2 v2:172.21.15.119:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 791ac3f1-f414-442d-8950-49875e1f19bf v742) v1 ==== 223+0+0 (secure 0 0 0) 0x7f05cc05ac60 con 0x7f05d80fc720 2024-09-10T19:18:16.526 INFO:tasks.workunit.client.0.smithi119.stderr:set-quota max_objects = 10 for pool 791ac3f1-f414-442d-8950-49875e1f19bf 2024-09-10T19:18:16.529 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.527+0000 7f05deab8640 1 -- 172.21.15.119:0/2232113587 >> v2:172.21.15.119:6800/3118961733 conn(0x7f05c806ef80 msgr2=0x7f05c8071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:16.529 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.527+0000 7f05deab8640 1 --2- 172.21.15.119:0/2232113587 >> v2:172.21.15.119:6800/3118961733 conn(0x7f05c806ef80 0x7f05c8071430 secure :-1 s=READY pgs=3706 cs=0 l=1 rev1=1 crypto rx=0x7f05d810e4f0 tx=0x7f05c4009da0 comp rx=0 tx=0).stop 2024-09-10T19:18:16.529 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.527+0000 7f05deab8640 1 -- 172.21.15.119:0/2232113587 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f05d80fc720 msgr2=0x7f05d8112be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:16.529 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.527+0000 7f05deab8640 1 --2- 172.21.15.119:0/2232113587 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f05d80fc720 0x7f05d8112be0 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7f05cc009780 tx=0x7f05cc009da0 comp rx=0 tx=0).stop 2024-09-10T19:18:16.529 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.527+0000 7f05deab8640 1 -- 172.21.15.119:0/2232113587 shutdown_connections 2024-09-10T19:18:16.529 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.527+0000 7f05deab8640 1 --2- 172.21.15.119:0/2232113587 >> v2:172.21.15.119:6800/3118961733 conn(0x7f05c806ef80 0x7f05c8071430 unknown :-1 s=CLOSED pgs=3706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:16.529 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.527+0000 7f05deab8640 1 --2- 172.21.15.119:0/2232113587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f05d8102740 0x7f05d810cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:16.530 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.527+0000 7f05deab8640 1 --2- 172.21.15.119:0/2232113587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05d80fd0d0 0x7f05d8113120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:16.531 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.527+0000 7f05deab8640 1 --2- 172.21.15.119:0/2232113587 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f05d80fc720 0x7f05d8112be0 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:16.531 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.527+0000 7f05deab8640 1 -- 172.21.15.119:0/2232113587 >> 172.21.15.119:0/2232113587 conn(0x7f05d80f82c0 msgr2=0x7f05d8100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:18:16.531 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.527+0000 7f05deab8640 1 -- 172.21.15.119:0/2232113587 shutdown_connections 2024-09-10T19:18:16.531 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.527+0000 7f05deab8640 1 -- 172.21.15.119:0/2232113587 wait complete. 2024-09-10T19:18:16.539 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd pool application enable 791ac3f1-f414-442d-8950-49875e1f19bf rados 2024-09-10T19:18:16.623 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.620+0000 7f03bad9d640 1 -- 172.21.15.119:0/2223270884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03b41015f0 msgr2=0x7f03b41019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:16.624 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.620+0000 7f03bad9d640 1 --2- 172.21.15.119:0/2223270884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03b41015f0 0x7f03b41019c0 secure :-1 s=READY pgs=2429 cs=0 l=1 rev1=1 crypto rx=0x7f03a8004950 tx=0x7f03a8009da0 comp rx=0 tx=0).stop 2024-09-10T19:18:16.624 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.620+0000 7f03bad9d640 1 -- 172.21.15.119:0/2223270884 shutdown_connections 2024-09-10T19:18:16.624 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.620+0000 7f03bad9d640 1 --2- 172.21.15.119:0/2223270884 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03b40fbd30 0x7f03b41083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:16.624 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.620+0000 7f03bad9d640 1 --2- 172.21.15.119:0/2223270884 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03b4101f90 0x7f03b40fb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:16.624 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.620+0000 7f03bad9d640 1 --2- 172.21.15.119:0/2223270884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03b41015f0 0x7f03b41019c0 unknown :-1 s=CLOSED pgs=2429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:16.624 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.620+0000 7f03bad9d640 1 -- 172.21.15.119:0/2223270884 >> 172.21.15.119:0/2223270884 conn(0x7f03b4072510 msgr2=0x7f03b40f9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:18:16.624 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.620+0000 7f03bad9d640 1 -- 172.21.15.119:0/2223270884 shutdown_connections 2024-09-10T19:18:16.624 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.620+0000 7f03bad9d640 1 -- 172.21.15.119:0/2223270884 wait complete. 2024-09-10T19:18:16.624 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.620+0000 7f03bad9d640 1 Processor -- start 2024-09-10T19:18:16.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.620+0000 7f03bad9d640 1 -- start start 2024-09-10T19:18:16.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03bad9d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03b40fbd30 0x7f03b4112be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:16.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03bad9d640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03b41015f0 0x7f03b4113120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:16.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03bad9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03b4101f90 0x7f03b4065c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:16.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03bad9d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b40662b0 con 0x7f03b4101f90 2024-09-10T19:18:16.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03bad9d640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b4066420 con 0x7f03b41015f0 2024-09-10T19:18:16.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03bad9d640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b40665e0 con 0x7f03b40fbd30 2024-09-10T19:18:16.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b8b39640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03b40fbd30 0x7f03b4112be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:16.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b8b39640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03b40fbd30 0x7f03b4112be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:60050/0 (socket says 172.21.15.119:60050) 2024-09-10T19:18:16.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b8b39640 1 -- 172.21.15.119:0/825776340 learned_addr learned my addr 172.21.15.119:0/825776340 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:18:16.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b933a640 1 --2- 172.21.15.119:0/825776340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03b4101f90 0x7f03b4065c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:16.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b3fff640 1 --2- 172.21.15.119:0/825776340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03b41015f0 0x7f03b4113120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:16.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b8b39640 1 -- 172.21.15.119:0/825776340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03b41015f0 msgr2=0x7f03b4113120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:16.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b8b39640 1 --2- 172.21.15.119:0/825776340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03b41015f0 0x7f03b4113120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:16.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b8b39640 1 -- 172.21.15.119:0/825776340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03b4101f90 msgr2=0x7f03b4065c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:16.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b8b39640 1 --2- 172.21.15.119:0/825776340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03b4101f90 0x7f03b4065c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:16.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b8b39640 1 -- 172.21.15.119:0/825776340 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a0005040 con 0x7f03b40fbd30 2024-09-10T19:18:16.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b933a640 1 --2- 172.21.15.119:0/825776340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03b4101f90 0x7f03b4065c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:18:16.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b8b39640 1 --2- 172.21.15.119:0/825776340 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03b40fbd30 0x7f03b4112be0 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f03a802d040 tx=0x7f03a8035040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:18:16.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b17fa640 1 -- 172.21.15.119:0/825776340 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a8009850 con 0x7f03b40fbd30 2024-09-10T19:18:16.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03bad9d640 1 -- 172.21.15.119:0/825776340 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a8005040 con 0x7f03b40fbd30 2024-09-10T19:18:16.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b17fa640 1 -- 172.21.15.119:0/825776340 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f03a802ab70 con 0x7f03b40fbd30 2024-09-10T19:18:16.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03b17fa640 1 -- 172.21.15.119:0/825776340 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a8033420 con 0x7f03b40fbd30 2024-09-10T19:18:16.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.621+0000 7f03bad9d640 1 -- 172.21.15.119:0/825776340 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b4119a00 con 0x7f03b40fbd30 2024-09-10T19:18:16.627 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.622+0000 7f03bad9d640 1 -- 172.21.15.119:0/825776340 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b40fcd70 con 0x7f03b40fbd30 2024-09-10T19:18:16.627 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.623+0000 7f03b17fa640 1 -- 172.21.15.119:0/825776340 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f03a8030020 con 0x7f03b40fbd30 2024-09-10T19:18:16.627 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.623+0000 7f03b17fa640 1 --2- 172.21.15.119:0/825776340 >> v2:172.21.15.119:6800/3118961733 conn(0x7f03a406ef80 0x7f03a4071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:16.627 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.623+0000 7f03b17fa640 1 -- 172.21.15.119:0/825776340 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(742..742 src has 1..742) v4 ==== 7842+0+0 (secure 0 0 0) 0x7f03a80aede0 con 0x7f03b40fbd30 2024-09-10T19:18:16.628 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.623+0000 7f03b3fff640 1 --2- 172.21.15.119:0/825776340 >> v2:172.21.15.119:6800/3118961733 conn(0x7f03a406ef80 0x7f03a4071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:16.628 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.624+0000 7f03b3fff640 1 --2- 172.21.15.119:0/825776340 >> v2:172.21.15.119:6800/3118961733 conn(0x7f03a406ef80 0x7f03a4071430 secure :-1 s=READY pgs=3707 cs=0 l=1 rev1=1 crypto rx=0x7f03b410e470 tx=0x7f03a000b040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:18:16.629 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.628+0000 7f03b17fa640 1 -- 172.21.15.119:0/825776340 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f03a807b810 con 0x7f03b40fbd30 2024-09-10T19:18:16.787 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:16.785+0000 7f03bad9d640 1 -- 172.21.15.119:0/825776340 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"} v 0) v1 -- 0x7f03b40654d0 con 0x7f03b40fbd30 2024-09-10T19:18:16.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:16 smithi119 ceph-mon[29577]: pgmap v1161: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:18:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:16 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2232113587' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:18:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:18:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:16 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]': finished 2024-09-10T19:18:16.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:16 smithi119 ceph-mon[29577]: osdmap e742: 8 total, 8 up, 8 in 2024-09-10T19:18:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:16 smithi119 ceph-mon[40674]: pgmap v1161: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:18:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:16 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2232113587' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:18:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:18:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:16 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]': finished 2024-09-10T19:18:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:16 smithi119 ceph-mon[40674]: osdmap e742: 8 total, 8 up, 8 in 2024-09-10T19:18:17.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:16 smithi194 ceph-mon[32464]: pgmap v1161: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:18:17.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:16 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2232113587' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:18:17.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:18:17.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:16 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "10"}]': finished 2024-09-10T19:18:17.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:16 smithi194 ceph-mon[32464]: osdmap e742: 8 total, 8 up, 8 in 2024-09-10T19:18:17.551 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:17.550+0000 7f03b17fa640 1 -- 172.21.15.119:0/825776340 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]=0 enabled application 'rados' on pool '791ac3f1-f414-442d-8950-49875e1f19bf' v743) v1 ==== 213+0+0 (secure 0 0 0) 0x7f03a807b1b0 con 0x7f03b40fbd30 2024-09-10T19:18:17.645 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:17.644+0000 7f03bad9d640 1 -- 172.21.15.119:0/825776340 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"} v 0) v1 -- 0x7f03b410aa40 con 0x7f03b40fbd30 2024-09-10T19:18:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:17 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/825776340' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]: dispatch 2024-09-10T19:18:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:17 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]: dispatch 2024-09-10T19:18:17.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:17.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:17 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/825776340' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]: dispatch 2024-09-10T19:18:17.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:17 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]: dispatch 2024-09-10T19:18:17.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:18.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:17 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/825776340' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]: dispatch 2024-09-10T19:18:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:17 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]: dispatch 2024-09-10T19:18:18.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:18.563 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:18.561+0000 7f03b17fa640 1 -- 172.21.15.119:0/825776340 <== mon.2 v2:172.21.15.119:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]=0 enabled application 'rados' on pool '791ac3f1-f414-442d-8950-49875e1f19bf' v744) v1 ==== 213+0+0 (secure 0 0 0) 0x7f03a807b390 con 0x7f03b40fbd30 2024-09-10T19:18:18.563 INFO:tasks.workunit.client.0.smithi119.stderr:enabled application 'rados' on pool '791ac3f1-f414-442d-8950-49875e1f19bf' 2024-09-10T19:18:18.566 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:18.564+0000 7f03bad9d640 1 -- 172.21.15.119:0/825776340 >> v2:172.21.15.119:6800/3118961733 conn(0x7f03a406ef80 msgr2=0x7f03a4071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:18.566 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:18.564+0000 7f03bad9d640 1 --2- 172.21.15.119:0/825776340 >> v2:172.21.15.119:6800/3118961733 conn(0x7f03a406ef80 0x7f03a4071430 secure :-1 s=READY pgs=3707 cs=0 l=1 rev1=1 crypto rx=0x7f03b410e470 tx=0x7f03a000b040 comp rx=0 tx=0).stop 2024-09-10T19:18:18.566 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:18.564+0000 7f03bad9d640 1 -- 172.21.15.119:0/825776340 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03b40fbd30 msgr2=0x7f03b4112be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:18.566 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:18.564+0000 7f03bad9d640 1 --2- 172.21.15.119:0/825776340 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03b40fbd30 0x7f03b4112be0 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7f03a802d040 tx=0x7f03a8035040 comp rx=0 tx=0).stop 2024-09-10T19:18:18.566 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:18.564+0000 7f03bad9d640 1 -- 172.21.15.119:0/825776340 shutdown_connections 2024-09-10T19:18:18.566 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:18.564+0000 7f03bad9d640 1 --2- 172.21.15.119:0/825776340 >> v2:172.21.15.119:6800/3118961733 conn(0x7f03a406ef80 0x7f03a4071430 unknown :-1 s=CLOSED pgs=3707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:18.566 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:18.564+0000 7f03bad9d640 1 --2- 172.21.15.119:0/825776340 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03b4101f90 0x7f03b4065c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:18.566 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:18.564+0000 7f03bad9d640 1 --2- 172.21.15.119:0/825776340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03b41015f0 0x7f03b4113120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:18.566 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:18.564+0000 7f03bad9d640 1 --2- 172.21.15.119:0/825776340 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f03b40fbd30 0x7f03b4112be0 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:18.566 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:18.564+0000 7f03bad9d640 1 -- 172.21.15.119:0/825776340 >> 172.21.15.119:0/825776340 conn(0x7f03b4072510 msgr2=0x7f03b40ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:18:18.566 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:18.564+0000 7f03bad9d640 1 -- 172.21.15.119:0/825776340 shutdown_connections 2024-09-10T19:18:18.567 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:18.564+0000 7f03bad9d640 1 -- 172.21.15.119:0/825776340 wait complete. 2024-09-10T19:18:18.576 INFO:tasks.workunit.client.0.smithi119.stderr:++ seq 1 10 2024-09-10T19:18:18.712 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:18:18.712 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put obj1 /etc/passwd 2024-09-10T19:18:18.746 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:18:18.746 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put obj2 /etc/passwd 2024-09-10T19:18:18.780 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:18:18.780 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put obj3 /etc/passwd 2024-09-10T19:18:18.811 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:18:18.811 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put obj4 /etc/passwd 2024-09-10T19:18:18.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:18 smithi119 ceph-mon[29577]: pgmap v1164: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:18:18.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]': finished 2024-09-10T19:18:18.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:18 smithi119 ceph-mon[29577]: osdmap e743: 8 total, 8 up, 8 in 2024-09-10T19:18:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:18 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/825776340' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]: dispatch 2024-09-10T19:18:18.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:18 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]: dispatch 2024-09-10T19:18:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:18 smithi119 ceph-mon[40674]: pgmap v1164: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:18:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]': finished 2024-09-10T19:18:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:18 smithi119 ceph-mon[40674]: osdmap e743: 8 total, 8 up, 8 in 2024-09-10T19:18:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:18 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/825776340' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]: dispatch 2024-09-10T19:18:18.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:18 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]: dispatch 2024-09-10T19:18:18.842 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:18:18.842 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put obj5 /etc/passwd 2024-09-10T19:18:18.873 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:18:18.873 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put obj6 /etc/passwd 2024-09-10T19:18:18.904 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:18:18.904 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put obj7 /etc/passwd 2024-09-10T19:18:18.920 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:18 smithi194 ceph-mon[32464]: pgmap v1164: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:18:18.920 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]': finished 2024-09-10T19:18:18.920 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:18 smithi194 ceph-mon[32464]: osdmap e743: 8 total, 8 up, 8 in 2024-09-10T19:18:18.920 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:18 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/825776340' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]: dispatch 2024-09-10T19:18:18.920 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:18 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]: dispatch 2024-09-10T19:18:18.934 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:18:18.934 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put obj8 /etc/passwd 2024-09-10T19:18:18.965 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:18:18.965 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put obj9 /etc/passwd 2024-09-10T19:18:18.995 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:18:18.995 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put obj10 /etc/passwd 2024-09-10T19:18:19.025 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 30 2024-09-10T19:18:19.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:18:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:18:19.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:19 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]': finished 2024-09-10T19:18:19.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:19 smithi119 ceph-mon[29577]: osdmap e744: 8 total, 8 up, 8 in 2024-09-10T19:18:19.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:19 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]': finished 2024-09-10T19:18:19.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:19 smithi119 ceph-mon[40674]: osdmap e744: 8 total, 8 up, 8 in 2024-09-10T19:18:20.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:19 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "app": "rados"}]': finished 2024-09-10T19:18:20.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:19 smithi194 ceph-mon[32464]: osdmap e744: 8 total, 8 up, 8 in 2024-09-10T19:18:20.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:18:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:18:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:18:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:20 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:20 smithi119 ceph-mon[29577]: pgmap v1167: 176 pgs: 176 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:20 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:18:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:20 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:20 smithi119 ceph-mon[40674]: pgmap v1167: 176 pgs: 176 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:20 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:18:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:20 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:20 smithi194 ceph-mon[32464]: pgmap v1167: 176 pgs: 176 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:20 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:18:23.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:22 smithi194 ceph-mon[32464]: pgmap v1168: 176 pgs: 176 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 882 B/s rd, 0 op/s 2024-09-10T19:18:23.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:22 smithi119 ceph-mon[29577]: pgmap v1168: 176 pgs: 176 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 882 B/s rd, 0 op/s 2024-09-10T19:18:23.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:22 smithi119 ceph-mon[40674]: pgmap v1168: 176 pgs: 176 active+clean; 582 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 882 B/s rd, 0 op/s 2024-09-10T19:18:23.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:18:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:18:22] "GET /metrics HTTP/1.1" 200 226143 "" "Prometheus/2.43.0" 2024-09-10T19:18:24.853 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:24 smithi119 ceph-mon[29577]: pgmap v1169: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 19 KiB/s wr, 2 op/s 2024-09-10T19:18:24.854 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:24 smithi119 ceph-mon[40674]: pgmap v1169: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 19 KiB/s wr, 2 op/s 2024-09-10T19:18:25.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:24 smithi194 ceph-mon[32464]: pgmap v1169: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 19 KiB/s wr, 2 op/s 2024-09-10T19:18:27.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:26 smithi194 ceph-mon[32464]: pgmap v1170: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 16 KiB/s wr, 2 op/s 2024-09-10T19:18:27.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:26 smithi119 ceph-mon[29577]: pgmap v1170: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 16 KiB/s wr, 2 op/s 2024-09-10T19:18:27.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:26 smithi119 ceph-mon[40674]: pgmap v1170: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 16 KiB/s wr, 2 op/s 2024-09-10T19:18:28.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:27 smithi194 ceph-mon[32464]: pool '791ac3f1-f414-442d-8950-49875e1f19bf' is full (reached quota's max_objects: 10) 2024-09-10T19:18:28.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:27 smithi194 ceph-mon[32464]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T19:18:28.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:27 smithi194 ceph-mon[32464]: osdmap e745: 8 total, 8 up, 8 in 2024-09-10T19:18:28.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:27 smithi119 ceph-mon[29577]: pool '791ac3f1-f414-442d-8950-49875e1f19bf' is full (reached quota's max_objects: 10) 2024-09-10T19:18:28.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:27 smithi119 ceph-mon[29577]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T19:18:28.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:27 smithi119 ceph-mon[29577]: osdmap e745: 8 total, 8 up, 8 in 2024-09-10T19:18:28.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:27 smithi119 ceph-mon[40674]: pool '791ac3f1-f414-442d-8950-49875e1f19bf' is full (reached quota's max_objects: 10) 2024-09-10T19:18:28.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:27 smithi119 ceph-mon[40674]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T19:18:28.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:27 smithi119 ceph-mon[40674]: osdmap e745: 8 total, 8 up, 8 in 2024-09-10T19:18:28.927 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:28 smithi194 ceph-mon[32464]: pgmap v1171: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-09-10T19:18:29.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:28 smithi119 ceph-mon[29577]: pgmap v1171: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-09-10T19:18:29.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:28 smithi119 ceph-mon[40674]: pgmap v1171: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 13 KiB/s wr, 2 op/s 2024-09-10T19:18:29.269 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:18:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:18:30.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:18:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:18:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:18:31.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:30 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:31.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:30 smithi194 ceph-mon[32464]: pgmap v1173: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:18:31.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:30 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:31.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:30 smithi119 ceph-mon[29577]: pgmap v1173: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:18:31.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:30 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:31.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:30 smithi119 ceph-mon[40674]: pgmap v1173: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:18:33.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:32 smithi194 ceph-mon[32464]: pgmap v1174: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:18:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:18:33.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:33.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:32 smithi119 ceph-mon[29577]: pgmap v1174: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:18:33.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:18:33.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:33.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:18:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:18:32] "GET /metrics HTTP/1.1" 200 226193 "" "Prometheus/2.43.0" 2024-09-10T19:18:33.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:32 smithi119 ceph-mon[40674]: pgmap v1174: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:18:33.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:18:33.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:34 smithi119 ceph-mon[29577]: pgmap v1175: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:18:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:34 smithi119 ceph-mon[40674]: pgmap v1175: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:18:35.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:34 smithi194 ceph-mon[32464]: pgmap v1175: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:18:37.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:36 smithi194 ceph-mon[32464]: pgmap v1176: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:18:37.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:36 smithi119 ceph-mon[29577]: pgmap v1176: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:18:37.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:36 smithi119 ceph-mon[40674]: pgmap v1176: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:18:38.938 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:38 smithi194 ceph-mon[32464]: pgmap v1177: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:18:39.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:38 smithi119 ceph-mon[29577]: pgmap v1177: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:18:39.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:38 smithi119 ceph-mon[40674]: pgmap v1177: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:18:39.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:18:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:18:40.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:18:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:18:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:18:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:40 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:41.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:40 smithi194 ceph-mon[32464]: pgmap v1178: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:40 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:41.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:40 smithi119 ceph-mon[29577]: pgmap v1178: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:40 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:41.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:40 smithi119 ceph-mon[40674]: pgmap v1178: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:43.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:42 smithi194 ceph-mon[32464]: pgmap v1179: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:18:43.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:42 smithi119 ceph-mon[29577]: pgmap v1179: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:18:43.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:18:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:18:42] "GET /metrics HTTP/1.1" 200 226193 "" "Prometheus/2.43.0" 2024-09-10T19:18:43.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:42 smithi119 ceph-mon[40674]: pgmap v1179: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:18:45.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:44 smithi194 ceph-mon[32464]: pgmap v1180: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:45.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:44 smithi119 ceph-mon[29577]: pgmap v1180: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:45.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:44 smithi119 ceph-mon[40674]: pgmap v1180: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:47.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:46 smithi194 ceph-mon[32464]: pgmap v1181: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:18:47.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:46 smithi119 ceph-mon[29577]: pgmap v1181: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:18:47.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:46 smithi119 ceph-mon[40674]: pgmap v1181: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:18:48.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:48.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:48.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:48.944 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:48 smithi194 ceph-mon[32464]: pgmap v1182: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:18:49.028 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=148497 2024-09-10T19:18:49.028 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd pool set-quota 791ac3f1-f414-442d-8950-49875e1f19bf max_objects 100 2024-09-10T19:18:49.028 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put onemore /etc/passwd 2024-09-10T19:18:49.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:48 smithi119 ceph-mon[29577]: pgmap v1182: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:18:49.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:48 smithi119 ceph-mon[40674]: pgmap v1182: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:18:49.119 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 -- 172.21.15.119:0/1264476678 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7010101f90 msgr2=0x7f70100fb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:49.119 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 --2- 172.21.15.119:0/1264476678 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7010101f90 0x7f70100fb650 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7f7004004950 tx=0x7f7004009da0 comp rx=0 tx=0).stop 2024-09-10T19:18:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 -- 172.21.15.119:0/1264476678 shutdown_connections 2024-09-10T19:18:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 --2- 172.21.15.119:0/1264476678 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f70100fbd30 0x7f70101083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 --2- 172.21.15.119:0/1264476678 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7010101f90 0x7f70100fb650 unknown :-1 s=CLOSED pgs=2945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 --2- 172.21.15.119:0/1264476678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70101015f0 0x7f70101019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 -- 172.21.15.119:0/1264476678 >> 172.21.15.119:0/1264476678 conn(0x7f7010072530 msgr2=0x7f70100f9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:18:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 -- 172.21.15.119:0/1264476678 shutdown_connections 2024-09-10T19:18:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 -- 172.21.15.119:0/1264476678 wait complete. 2024-09-10T19:18:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 Processor -- start 2024-09-10T19:18:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 -- start start 2024-09-10T19:18:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f70100fbd30 0x7f7010112bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:49.120 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70101015f0 0x7f7010113130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:49.121 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7010101f90 0x7f7010064a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:49.121 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70100650b0 con 0x7f70101015f0 2024-09-10T19:18:49.121 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7010065220 con 0x7f70100fbd30 2024-09-10T19:18:49.121 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.115+0000 7f7014d9d640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70100653e0 con 0x7f7010101f90 2024-09-10T19:18:49.121 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f700dd9b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70101015f0 0x7f7010113130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:49.121 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f700dd9b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70101015f0 0x7f7010113130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57730/0 (socket says 172.21.15.119:57730) 2024-09-10T19:18:49.121 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f700ed9d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7010101f90 0x7f7010064a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:49.121 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f700dd9b640 1 -- 172.21.15.119:0/856970599 learned_addr learned my addr 172.21.15.119:0/856970599 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:18:49.121 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f700e59c640 1 --2- 172.21.15.119:0/856970599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f70100fbd30 0x7f7010112bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:49.121 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f700dd9b640 1 -- 172.21.15.119:0/856970599 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7010101f90 msgr2=0x7f7010064a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:49.122 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f700dd9b640 1 --2- 172.21.15.119:0/856970599 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7010101f90 0x7f7010064a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:49.122 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f700dd9b640 1 -- 172.21.15.119:0/856970599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f70100fbd30 msgr2=0x7f7010112bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:49.122 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f700dd9b640 1 --2- 172.21.15.119:0/856970599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f70100fbd30 0x7f7010112bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:49.122 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f700dd9b640 1 -- 172.21.15.119:0/856970599 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7000005040 con 0x7f70101015f0 2024-09-10T19:18:49.122 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f700ed9d640 1 --2- 172.21.15.119:0/856970599 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7010101f90 0x7f7010064a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:18:49.122 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f700dd9b640 1 --2- 172.21.15.119:0/856970599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70101015f0 0x7f7010113130 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f700402d040 tx=0x7f7004035040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:18:49.122 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f6ffeffd640 1 -- 172.21.15.119:0/856970599 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700402a960 con 0x7f70101015f0 2024-09-10T19:18:49.122 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f7014d9d640 1 -- 172.21.15.119:0/856970599 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7004005040 con 0x7f70101015f0 2024-09-10T19:18:49.122 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f6ffeffd640 1 -- 172.21.15.119:0/856970599 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f70040325e0 con 0x7f70101015f0 2024-09-10T19:18:49.122 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f6ffeffd640 1 -- 172.21.15.119:0/856970599 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7004033690 con 0x7f70101015f0 2024-09-10T19:18:49.122 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.116+0000 7f7014d9d640 1 -- 172.21.15.119:0/856970599 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7010119a10 con 0x7f70101015f0 2024-09-10T19:18:49.122 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.117+0000 7f7014d9d640 1 -- 172.21.15.119:0/856970599 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70100fcd70 con 0x7f70101015f0 2024-09-10T19:18:49.122 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.118+0000 7f6ffeffd640 1 -- 172.21.15.119:0/856970599 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f7004030020 con 0x7f70101015f0 2024-09-10T19:18:49.123 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.118+0000 7f6ffeffd640 1 --2- 172.21.15.119:0/856970599 >> v2:172.21.15.119:6800/3118961733 conn(0x7f6ff806ef80 0x7f6ff8071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:49.123 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.118+0000 7f6ffeffd640 1 -- 172.21.15.119:0/856970599 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(745..745 src has 1..745) v4 ==== 7855+0+0 (secure 0 0 0) 0x7f70040aee40 con 0x7f70101015f0 2024-09-10T19:18:49.123 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.118+0000 7f700e59c640 1 --2- 172.21.15.119:0/856970599 >> v2:172.21.15.119:6800/3118961733 conn(0x7f6ff806ef80 0x7f6ff8071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:49.123 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.118+0000 7f6ffeffd640 1 -- 172.21.15.119:0/856970599 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=746}) v3 -- 0x7f6ff807a340 con 0x7f70101015f0 2024-09-10T19:18:49.123 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.119+0000 7f700e59c640 1 --2- 172.21.15.119:0/856970599 >> v2:172.21.15.119:6800/3118961733 conn(0x7f6ff806ef80 0x7f6ff8071430 secure :-1 s=READY pgs=3719 cs=0 l=1 rev1=1 crypto rx=0x7f7000007fd0 tx=0x7f7000009da0 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:18:49.124 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.123+0000 7f6ffeffd640 1 -- 172.21.15.119:0/856970599 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f700407b750 con 0x7f70101015f0 2024-09-10T19:18:49.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:18:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:18:49.279 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.278+0000 7f7014d9d640 1 -- 172.21.15.119:0/856970599 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f7010065620 con 0x7f70101015f0 2024-09-10T19:18:49.620 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.618+0000 7f6ffeffd640 1 -- 172.21.15.119:0/856970599 <== mon.0 v2:172.21.15.119:3300/0 7 ==== osd_map(746..746 src has 1..746) v4 ==== 620+0+0 (secure 0 0 0) 0x7f700407b490 con 0x7f70101015f0 2024-09-10T19:18:49.620 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.618+0000 7f6ffeffd640 1 -- 172.21.15.119:0/856970599 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=747}) v3 -- 0x7f6ff807af10 con 0x7f70101015f0 2024-09-10T19:18:49.623 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.622+0000 7f6ffeffd640 1 -- 172.21.15.119:0/856970599 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 791ac3f1-f414-442d-8950-49875e1f19bf v746) v1 ==== 225+0+0 (secure 0 0 0) 0x7f700407b0f0 con 0x7f70101015f0 2024-09-10T19:18:49.718 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:49.717+0000 7f7014d9d640 1 -- 172.21.15.119:0/856970599 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7f701004bfe0 con 0x7f70101015f0 2024-09-10T19:18:50.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:49 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/856970599' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"}]: dispatch 2024-09-10T19:18:50.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:49 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/856970599' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"}]: dispatch 2024-09-10T19:18:50.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:49 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/856970599' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"}]: dispatch 2024-09-10T19:18:50.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:18:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:18:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:18:50.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.624+0000 7f6ffeffd640 1 -- 172.21.15.119:0/856970599 <== mon.0 v2:172.21.15.119:3300/0 9 ==== osd_map(747..747 src has 1..747) v4 ==== 620+0+0 (secure 0 0 0) 0x7f700407c080 con 0x7f70101015f0 2024-09-10T19:18:50.626 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.624+0000 7f6ffeffd640 1 -- 172.21.15.119:0/856970599 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=748}) v3 -- 0x7f6ff807b370 con 0x7f70101015f0 2024-09-10T19:18:50.628 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.627+0000 7f6ffeffd640 1 -- 172.21.15.119:0/856970599 <== mon.0 v2:172.21.15.119:3300/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 791ac3f1-f414-442d-8950-49875e1f19bf v747) v1 ==== 225+0+0 (secure 0 0 0) 0x7f700407b2d0 con 0x7f70101015f0 2024-09-10T19:18:50.628 INFO:tasks.workunit.client.0.smithi119.stderr:set-quota max_objects = 100 for pool 791ac3f1-f414-442d-8950-49875e1f19bf 2024-09-10T19:18:50.633 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.631+0000 7f7014d9d640 1 -- 172.21.15.119:0/856970599 >> v2:172.21.15.119:6800/3118961733 conn(0x7f6ff806ef80 msgr2=0x7f6ff8071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:50.633 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.631+0000 7f7014d9d640 1 --2- 172.21.15.119:0/856970599 >> v2:172.21.15.119:6800/3118961733 conn(0x7f6ff806ef80 0x7f6ff8071430 secure :-1 s=READY pgs=3719 cs=0 l=1 rev1=1 crypto rx=0x7f7000007fd0 tx=0x7f7000009da0 comp rx=0 tx=0).stop 2024-09-10T19:18:50.633 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.631+0000 7f7014d9d640 1 -- 172.21.15.119:0/856970599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70101015f0 msgr2=0x7f7010113130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:50.633 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.631+0000 7f7014d9d640 1 --2- 172.21.15.119:0/856970599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70101015f0 0x7f7010113130 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f700402d040 tx=0x7f7004035040 comp rx=0 tx=0).stop 2024-09-10T19:18:50.633 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.631+0000 7f7014d9d640 1 -- 172.21.15.119:0/856970599 shutdown_connections 2024-09-10T19:18:50.634 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.631+0000 7f7014d9d640 1 --2- 172.21.15.119:0/856970599 >> v2:172.21.15.119:6800/3118961733 conn(0x7f6ff806ef80 0x7f6ff8071430 unknown :-1 s=CLOSED pgs=3719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:50.634 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.631+0000 7f7014d9d640 1 --2- 172.21.15.119:0/856970599 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f7010101f90 0x7f7010064a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:50.634 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.631+0000 7f7014d9d640 1 --2- 172.21.15.119:0/856970599 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70101015f0 0x7f7010113130 unknown :-1 s=CLOSED pgs=2446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:50.634 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.631+0000 7f7014d9d640 1 --2- 172.21.15.119:0/856970599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f70100fbd30 0x7f7010112bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:50.634 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.631+0000 7f7014d9d640 1 -- 172.21.15.119:0/856970599 >> 172.21.15.119:0/856970599 conn(0x7f7010072530 msgr2=0x7f70100ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:18:50.634 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.631+0000 7f7014d9d640 1 -- 172.21.15.119:0/856970599 shutdown_connections 2024-09-10T19:18:50.634 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:50.631+0000 7f7014d9d640 1 -- 172.21.15.119:0/856970599 wait complete. 2024-09-10T19:18:50.644 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 148497 2024-09-10T19:18:51.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:50 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:51.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:50 smithi194 ceph-mon[32464]: pgmap v1183: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:51.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/856970599' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"}]': finished 2024-09-10T19:18:51.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:50 smithi194 ceph-mon[32464]: osdmap e746: 8 total, 8 up, 8 in 2024-09-10T19:18:51.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:50 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/856970599' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"}]: dispatch 2024-09-10T19:18:51.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:50 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:51.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:50 smithi119 ceph-mon[29577]: pgmap v1183: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:51.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/856970599' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"}]': finished 2024-09-10T19:18:51.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:50 smithi119 ceph-mon[29577]: osdmap e746: 8 total, 8 up, 8 in 2024-09-10T19:18:51.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:50 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/856970599' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"}]: dispatch 2024-09-10T19:18:51.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:50 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:51.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:50 smithi119 ceph-mon[40674]: pgmap v1183: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:18:51.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/856970599' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"}]': finished 2024-09-10T19:18:51.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:50 smithi119 ceph-mon[40674]: osdmap e746: 8 total, 8 up, 8 in 2024-09-10T19:18:51.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:50 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/856970599' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"}]: dispatch 2024-09-10T19:18:52.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:51 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/856970599' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"}]': finished 2024-09-10T19:18:52.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:51 smithi194 ceph-mon[32464]: osdmap e747: 8 total, 8 up, 8 in 2024-09-10T19:18:52.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:51 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/856970599' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"}]': finished 2024-09-10T19:18:52.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:51 smithi119 ceph-mon[29577]: osdmap e747: 8 total, 8 up, 8 in 2024-09-10T19:18:52.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:51 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/856970599' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "100"}]': finished 2024-09-10T19:18:52.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:51 smithi119 ceph-mon[40674]: osdmap e747: 8 total, 8 up, 8 in 2024-09-10T19:18:52.347 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 0 -ne 0 ']' 2024-09-10T19:18:52.347 INFO:tasks.workunit.client.0.smithi119.stderr:+ true 2024-09-10T19:18:52.347 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put twomore /etc/passwd 2024-09-10T19:18:52.382 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd pool set-quota 791ac3f1-f414-442d-8950-49875e1f19bf max_bytes 100 2024-09-10T19:18:52.465 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.461+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/3753019249 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3ccc1015f0 msgr2=0x7f3ccc1019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:52.465 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.461+0000 7f3cd0a8b640 1 --2- 172.21.15.119:0/3753019249 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3ccc1015f0 0x7f3ccc1019c0 secure :-1 s=READY pgs=2946 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc004950 tx=0x7f3cbc009da0 comp rx=0 tx=0).stop 2024-09-10T19:18:52.465 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.461+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/3753019249 shutdown_connections 2024-09-10T19:18:52.465 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.461+0000 7f3cd0a8b640 1 --2- 172.21.15.119:0/3753019249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ccc0fbd30 0x7f3ccc1083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:52.465 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.461+0000 7f3cd0a8b640 1 --2- 172.21.15.119:0/3753019249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ccc101f90 0x7f3ccc0fb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:52.465 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.461+0000 7f3cd0a8b640 1 --2- 172.21.15.119:0/3753019249 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3ccc1015f0 0x7f3ccc1019c0 unknown :-1 s=CLOSED pgs=2946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:52.465 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.461+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/3753019249 >> 172.21.15.119:0/3753019249 conn(0x7f3ccc0725d0 msgr2=0x7f3ccc0f9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:18:52.465 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.461+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/3753019249 shutdown_connections 2024-09-10T19:18:52.466 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.461+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/3753019249 wait complete. 2024-09-10T19:18:52.466 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.462+0000 7f3cd0a8b640 1 Processor -- start 2024-09-10T19:18:52.466 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.462+0000 7f3cd0a8b640 1 -- start start 2024-09-10T19:18:52.466 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.462+0000 7f3cd0a8b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ccc0fbd30 0x7f3ccc10dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:52.466 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.462+0000 7f3cd0a8b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3ccc1015f0 0x7f3ccc10e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:52.466 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.462+0000 7f3cd0a8b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ccc101f90 0x7f3ccc10e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:52.466 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.462+0000 7f3cd0a8b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc119060 con 0x7f3ccc0fbd30 2024-09-10T19:18:52.466 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.462+0000 7f3cd0a8b640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc1191d0 con 0x7f3ccc101f90 2024-09-10T19:18:52.466 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.462+0000 7f3cd0a8b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc119390 con 0x7f3ccc1015f0 2024-09-10T19:18:52.466 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.462+0000 7f3cc1d9b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3ccc1015f0 0x7f3ccc10e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:52.466 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.462+0000 7f3cca59c640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ccc0fbd30 0x7f3ccc10dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:52.466 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.462+0000 7f3cc1d9b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3ccc1015f0 0x7f3ccc10e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:51110/0 (socket says 172.21.15.119:51110) 2024-09-10T19:18:52.466 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.462+0000 7f3cc1d9b640 1 -- 172.21.15.119:0/1144926814 learned_addr learned my addr 172.21.15.119:0/1144926814 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:18:52.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.463+0000 7f3cca59c640 1 -- 172.21.15.119:0/1144926814 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3ccc1015f0 msgr2=0x7f3ccc10e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:52.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.463+0000 7f3cca59c640 1 --2- 172.21.15.119:0/1144926814 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3ccc1015f0 0x7f3ccc10e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:52.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.463+0000 7f3cca59c640 1 -- 172.21.15.119:0/1144926814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ccc101f90 msgr2=0x7f3ccc10e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:52.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.463+0000 7f3cca59c640 1 --2- 172.21.15.119:0/1144926814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ccc101f90 0x7f3ccc10e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:52.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.463+0000 7f3cca59c640 1 -- 172.21.15.119:0/1144926814 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc40098b0 con 0x7f3ccc0fbd30 2024-09-10T19:18:52.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.463+0000 7f3cc1d9b640 1 --2- 172.21.15.119:0/1144926814 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3ccc1015f0 0x7f3ccc10e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:18:52.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.463+0000 7f3cca59c640 1 --2- 172.21.15.119:0/1144926814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ccc0fbd30 0x7f3ccc10dd90 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc02a480 tx=0x7f3cbc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:18:52.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.463+0000 7f3cc37fe640 1 -- 172.21.15.119:0/1144926814 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc02ab50 con 0x7f3ccc0fbd30 2024-09-10T19:18:52.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.463+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/1144926814 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cbc005040 con 0x7f3ccc0fbd30 2024-09-10T19:18:52.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.463+0000 7f3cc37fe640 1 -- 172.21.15.119:0/1144926814 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f3cbc02acb0 con 0x7f3ccc0fbd30 2024-09-10T19:18:52.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.463+0000 7f3cc37fe640 1 -- 172.21.15.119:0/1144926814 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc0323d0 con 0x7f3ccc0fbd30 2024-09-10T19:18:52.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.463+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/1144926814 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ccc1199a0 con 0x7f3ccc0fbd30 2024-09-10T19:18:52.468 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.464+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/1144926814 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ccc0fcd70 con 0x7f3ccc0fbd30 2024-09-10T19:18:52.468 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.464+0000 7f3cc37fe640 1 -- 172.21.15.119:0/1144926814 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f3cbc030020 con 0x7f3ccc0fbd30 2024-09-10T19:18:52.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.465+0000 7f3cc37fe640 1 --2- 172.21.15.119:0/1144926814 >> v2:172.21.15.119:6800/3118961733 conn(0x7f3cb806ef80 0x7f3cb8071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:18:52.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.465+0000 7f3cc37fe640 1 -- 172.21.15.119:0/1144926814 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(748..748 src has 1..748) v4 ==== 7855+0+0 (secure 0 0 0) 0x7f3cbc0ae290 con 0x7f3ccc0fbd30 2024-09-10T19:18:52.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.465+0000 7f3cc1d9b640 1 --2- 172.21.15.119:0/1144926814 >> v2:172.21.15.119:6800/3118961733 conn(0x7f3cb806ef80 0x7f3cb8071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:18:52.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.465+0000 7f3cc1d9b640 1 --2- 172.21.15.119:0/1144926814 >> v2:172.21.15.119:6800/3118961733 conn(0x7f3cb806ef80 0x7f3cb8071430 secure :-1 s=READY pgs=3721 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0664e0 tx=0x7f3cb400b040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:18:52.470 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.469+0000 7f3cc37fe640 1 -- 172.21.15.119:0/1144926814 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f3cbc07ab90 con 0x7f3ccc0fbd30 2024-09-10T19:18:52.625 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:52.624+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/1144926814 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7f3ccc0662c0 con 0x7f3ccc0fbd30 2024-09-10T19:18:52.968 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:18:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:18:52] "GET /metrics HTTP/1.1" 200 226173 "" "Prometheus/2.43.0" 2024-09-10T19:18:52.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:52 smithi119 ceph-mon[29577]: pgmap v1186: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:18:52.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:52 smithi119 ceph-mon[29577]: pool '791ac3f1-f414-442d-8950-49875e1f19bf' no longer out of quota; removing NO_QUOTA flag 2024-09-10T19:18:52.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:52 smithi119 ceph-mon[29577]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T19:18:52.969 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:52 smithi119 ceph-mon[29577]: osdmap e748: 8 total, 8 up, 8 in 2024-09-10T19:18:52.970 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:52 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1144926814' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"}]: dispatch 2024-09-10T19:18:52.970 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:52 smithi119 ceph-mon[40674]: pgmap v1186: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:18:52.970 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:52 smithi119 ceph-mon[40674]: pool '791ac3f1-f414-442d-8950-49875e1f19bf' no longer out of quota; removing NO_QUOTA flag 2024-09-10T19:18:52.970 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:52 smithi119 ceph-mon[40674]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T19:18:52.970 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:52 smithi119 ceph-mon[40674]: osdmap e748: 8 total, 8 up, 8 in 2024-09-10T19:18:52.971 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:52 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1144926814' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"}]: dispatch 2024-09-10T19:18:53.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:52 smithi194 ceph-mon[32464]: pgmap v1186: 176 pgs: 176 active+clean; 707 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:18:53.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:52 smithi194 ceph-mon[32464]: pool '791ac3f1-f414-442d-8950-49875e1f19bf' no longer out of quota; removing NO_QUOTA flag 2024-09-10T19:18:53.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:52 smithi194 ceph-mon[32464]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T19:18:53.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:52 smithi194 ceph-mon[32464]: osdmap e748: 8 total, 8 up, 8 in 2024-09-10T19:18:53.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:52 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1144926814' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"}]: dispatch 2024-09-10T19:18:53.340 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:53.338+0000 7f3cc37fe640 1 -- 172.21.15.119:0/1144926814 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 791ac3f1-f414-442d-8950-49875e1f19bf v749) v1 ==== 221+0+0 (secure 0 0 0) 0x7f3cbc07a530 con 0x7f3ccc0fbd30 2024-09-10T19:18:53.436 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:53.434+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/1144926814 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7f3ccc119bc0 con 0x7f3ccc0fbd30 2024-09-10T19:18:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:53 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:18:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:53 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:18:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:53 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:18:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:53 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:18:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1144926814' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"}]': finished 2024-09-10T19:18:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:53 smithi194 ceph-mon[32464]: osdmap e749: 8 total, 8 up, 8 in 2024-09-10T19:18:54.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:53 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1144926814' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"}]: dispatch 2024-09-10T19:18:54.066 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:53 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:18:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:53 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:18:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:53 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:18:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:53 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:18:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1144926814' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"}]': finished 2024-09-10T19:18:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:53 smithi119 ceph-mon[40674]: osdmap e749: 8 total, 8 up, 8 in 2024-09-10T19:18:54.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:53 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1144926814' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"}]: dispatch 2024-09-10T19:18:54.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:53 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:18:54.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:53 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:18:54.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:53 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:18:54.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:53 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:18:54.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1144926814' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"}]': finished 2024-09-10T19:18:54.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:53 smithi119 ceph-mon[29577]: osdmap e749: 8 total, 8 up, 8 in 2024-09-10T19:18:54.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:53 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1144926814' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"}]: dispatch 2024-09-10T19:18:54.342 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:54.340+0000 7f3cc37fe640 1 -- 172.21.15.119:0/1144926814 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 791ac3f1-f414-442d-8950-49875e1f19bf v750) v1 ==== 221+0+0 (secure 0 0 0) 0x7f3cbc07a710 con 0x7f3ccc0fbd30 2024-09-10T19:18:54.342 INFO:tasks.workunit.client.0.smithi119.stderr:set-quota max_bytes = 100 for pool 791ac3f1-f414-442d-8950-49875e1f19bf 2024-09-10T19:18:54.345 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:54.343+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/1144926814 >> v2:172.21.15.119:6800/3118961733 conn(0x7f3cb806ef80 msgr2=0x7f3cb8071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:54.345 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:54.343+0000 7f3cd0a8b640 1 --2- 172.21.15.119:0/1144926814 >> v2:172.21.15.119:6800/3118961733 conn(0x7f3cb806ef80 0x7f3cb8071430 secure :-1 s=READY pgs=3721 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc0664e0 tx=0x7f3cb400b040 comp rx=0 tx=0).stop 2024-09-10T19:18:54.345 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:54.343+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/1144926814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ccc0fbd30 msgr2=0x7f3ccc10dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:18:54.345 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:54.343+0000 7f3cd0a8b640 1 --2- 172.21.15.119:0/1144926814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ccc0fbd30 0x7f3ccc10dd90 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc02a480 tx=0x7f3cbc009da0 comp rx=0 tx=0).stop 2024-09-10T19:18:54.345 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:54.343+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/1144926814 shutdown_connections 2024-09-10T19:18:54.345 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:54.343+0000 7f3cd0a8b640 1 --2- 172.21.15.119:0/1144926814 >> v2:172.21.15.119:6800/3118961733 conn(0x7f3cb806ef80 0x7f3cb8071430 unknown :-1 s=CLOSED pgs=3721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:54.345 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:54.343+0000 7f3cd0a8b640 1 --2- 172.21.15.119:0/1144926814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ccc101f90 0x7f3ccc10e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:54.345 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:54.343+0000 7f3cd0a8b640 1 --2- 172.21.15.119:0/1144926814 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3ccc1015f0 0x7f3ccc10e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:54.345 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:54.343+0000 7f3cd0a8b640 1 --2- 172.21.15.119:0/1144926814 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ccc0fbd30 0x7f3ccc10dd90 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:18:54.345 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:54.343+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/1144926814 >> 172.21.15.119:0/1144926814 conn(0x7f3ccc0725d0 msgr2=0x7f3ccc0ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:18:54.345 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:54.343+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/1144926814 shutdown_connections 2024-09-10T19:18:54.346 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:18:54.343+0000 7f3cd0a8b640 1 -- 172.21.15.119:0/1144926814 wait complete. 2024-09-10T19:18:54.355 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 30 2024-09-10T19:18:55.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:54 smithi194 ceph-mon[32464]: pgmap v1188: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.3 KiB/s wr, 1 op/s 2024-09-10T19:18:55.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:54 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1144926814' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"}]': finished 2024-09-10T19:18:55.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:54 smithi194 ceph-mon[32464]: osdmap e750: 8 total, 8 up, 8 in 2024-09-10T19:18:55.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:54 smithi119 ceph-mon[29577]: pgmap v1188: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.3 KiB/s wr, 1 op/s 2024-09-10T19:18:55.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:54 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1144926814' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"}]': finished 2024-09-10T19:18:55.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:54 smithi119 ceph-mon[29577]: osdmap e750: 8 total, 8 up, 8 in 2024-09-10T19:18:55.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:54 smithi119 ceph-mon[40674]: pgmap v1188: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 4.3 KiB/s wr, 1 op/s 2024-09-10T19:18:55.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:54 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1144926814' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "100"}]': finished 2024-09-10T19:18:55.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:54 smithi119 ceph-mon[40674]: osdmap e750: 8 total, 8 up, 8 in 2024-09-10T19:18:57.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:56 smithi194 ceph-mon[32464]: pgmap v1191: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 5.5 KiB/s wr, 1 op/s 2024-09-10T19:18:57.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:56 smithi119 ceph-mon[29577]: pgmap v1191: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 5.5 KiB/s wr, 1 op/s 2024-09-10T19:18:57.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:56 smithi119 ceph-mon[40674]: pgmap v1191: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 5.5 KiB/s wr, 1 op/s 2024-09-10T19:18:58.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:57 smithi194 ceph-mon[32464]: pool '791ac3f1-f414-442d-8950-49875e1f19bf' is full (reached quota's max_bytes: 100 B) 2024-09-10T19:18:58.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:57 smithi194 ceph-mon[32464]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T19:18:58.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:57 smithi194 ceph-mon[32464]: osdmap e751: 8 total, 8 up, 8 in 2024-09-10T19:18:58.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:57 smithi119 ceph-mon[29577]: pool '791ac3f1-f414-442d-8950-49875e1f19bf' is full (reached quota's max_bytes: 100 B) 2024-09-10T19:18:58.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:57 smithi119 ceph-mon[29577]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T19:18:58.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:57 smithi119 ceph-mon[29577]: osdmap e751: 8 total, 8 up, 8 in 2024-09-10T19:18:58.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:57 smithi119 ceph-mon[40674]: pool '791ac3f1-f414-442d-8950-49875e1f19bf' is full (reached quota's max_bytes: 100 B) 2024-09-10T19:18:58.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:57 smithi119 ceph-mon[40674]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T19:18:58.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:57 smithi119 ceph-mon[40674]: osdmap e751: 8 total, 8 up, 8 in 2024-09-10T19:18:58.955 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:18:58 smithi194 ceph-mon[32464]: pgmap v1192: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 4.3 KiB/s wr, 1 op/s 2024-09-10T19:18:59.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:18:58 smithi119 ceph-mon[29577]: pgmap v1192: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 4.3 KiB/s wr, 1 op/s 2024-09-10T19:18:59.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:18:58 smithi119 ceph-mon[40674]: pgmap v1192: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 4.3 KiB/s wr, 1 op/s 2024-09-10T19:18:59.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:18:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:19:00.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:19:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:19:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:19:01.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:00 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:01.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:00 smithi194 ceph-mon[32464]: pgmap v1194: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:01.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:00 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:01.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:00 smithi119 ceph-mon[29577]: pgmap v1194: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:01.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:00 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:01.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:00 smithi119 ceph-mon[40674]: pgmap v1194: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:03.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:02 smithi194 ceph-mon[32464]: pgmap v1195: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 640 B/s rd, 0 op/s 2024-09-10T19:19:03.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:03.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:02 smithi119 ceph-mon[29577]: pgmap v1195: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 640 B/s rd, 0 op/s 2024-09-10T19:19:03.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:03.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:19:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:19:02] "GET /metrics HTTP/1.1" 200 226178 "" "Prometheus/2.43.0" 2024-09-10T19:19:03.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:02 smithi119 ceph-mon[40674]: pgmap v1195: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 640 B/s rd, 0 op/s 2024-09-10T19:19:03.068 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:05.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:04 smithi194 ceph-mon[32464]: pgmap v1196: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:19:05.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:04 smithi119 ceph-mon[29577]: pgmap v1196: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:19:05.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:04 smithi119 ceph-mon[40674]: pgmap v1196: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:19:07.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:06 smithi194 ceph-mon[32464]: pgmap v1197: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:19:07.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:06 smithi119 ceph-mon[29577]: pgmap v1197: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:19:07.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:06 smithi119 ceph-mon[40674]: pgmap v1197: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:19:08.966 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:08 smithi194 ceph-mon[32464]: pgmap v1198: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:19:09.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:08 smithi119 ceph-mon[29577]: pgmap v1198: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:19:09.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:08 smithi119 ceph-mon[40674]: pgmap v1198: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:19:09.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:19:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:19:10.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:19:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:19:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:19:11.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:10 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:11.020 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:10 smithi194 ceph-mon[32464]: pgmap v1199: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:11.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:10 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:11.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:10 smithi119 ceph-mon[29577]: pgmap v1199: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:11.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:10 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:11.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:10 smithi119 ceph-mon[40674]: pgmap v1199: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:13.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:12 smithi194 ceph-mon[32464]: pgmap v1200: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:13.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:12 smithi119 ceph-mon[29577]: pgmap v1200: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:13.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:12 smithi119 ceph-mon[40674]: pgmap v1200: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:13.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:19:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:19:12] "GET /metrics HTTP/1.1" 200 226178 "" "Prometheus/2.43.0" 2024-09-10T19:19:15.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:14 smithi194 ceph-mon[32464]: pgmap v1201: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:15.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:14 smithi119 ceph-mon[29577]: pgmap v1201: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:15.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:14 smithi119 ceph-mon[40674]: pgmap v1201: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:17.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:16 smithi194 ceph-mon[32464]: pgmap v1202: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:17.066 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:16 smithi119 ceph-mon[29577]: pgmap v1202: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:17.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:16 smithi119 ceph-mon[40674]: pgmap v1202: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:18.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:17 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:18.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:17 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:18.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:17 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:18.977 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:18 smithi194 ceph-mon[32464]: pgmap v1203: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:19.068 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:18 smithi119 ceph-mon[29577]: pgmap v1203: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:19.069 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:18 smithi119 ceph-mon[40674]: pgmap v1203: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:19.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:19:18 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:19:20.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:19:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:19:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:19:21.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:20 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:21.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:20 smithi194 ceph-mon[32464]: pgmap v1204: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:20 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:21.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:20 smithi119 ceph-mon[29577]: pgmap v1204: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:21.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:20 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:21.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:20 smithi119 ceph-mon[40674]: pgmap v1204: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:23.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:22 smithi194 ceph-mon[32464]: pgmap v1205: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:23.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:22 smithi119 ceph-mon[29577]: pgmap v1205: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:23.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:19:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:19:22] "GET /metrics HTTP/1.1" 200 226161 "" "Prometheus/2.43.0" 2024-09-10T19:19:23.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:22 smithi119 ceph-mon[40674]: pgmap v1205: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:24.357 INFO:tasks.workunit.client.0.smithi119.stderr:+ pid=148713 2024-09-10T19:19:24.357 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd pool set-quota 791ac3f1-f414-442d-8950-49875e1f19bf max_bytes 0 2024-09-10T19:19:24.358 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put two /etc/passwd 2024-09-10T19:19:24.447 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.445+0000 7f0384234640 1 -- 172.21.15.119:0/877999154 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f037c0fb680 msgr2=0x7f037c109530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:24.447 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.445+0000 7f0384234640 1 --2- 172.21.15.119:0/877999154 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f037c0fb680 0x7f037c109530 secure :-1 s=READY pgs=2949 cs=0 l=1 rev1=1 crypto rx=0x7f0374004980 tx=0x7f0374009da0 comp rx=0 tx=0).stop 2024-09-10T19:19:24.447 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.445+0000 7f0384234640 1 -- 172.21.15.119:0/877999154 shutdown_connections 2024-09-10T19:19:24.447 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.445+0000 7f0384234640 1 --2- 172.21.15.119:0/877999154 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f037c0fb680 0x7f037c109530 unknown :-1 s=CLOSED pgs=2949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:24.447 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.445+0000 7f0384234640 1 --2- 172.21.15.119:0/877999154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f037c0facd0 0x7f037c0fb140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:24.448 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.445+0000 7f0384234640 1 --2- 172.21.15.119:0/877999154 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f037c102780 0x7f037c102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:24.448 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.445+0000 7f0384234640 1 -- 172.21.15.119:0/877999154 >> 172.21.15.119:0/877999154 conn(0x7f037c074b80 msgr2=0x7f037c0f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:19:24.448 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.445+0000 7f0384234640 1 -- 172.21.15.119:0/877999154 shutdown_connections 2024-09-10T19:19:24.448 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.445+0000 7f0384234640 1 -- 172.21.15.119:0/877999154 wait complete. 2024-09-10T19:19:24.448 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f0384234640 1 Processor -- start 2024-09-10T19:19:24.448 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f0384234640 1 -- start start 2024-09-10T19:19:24.450 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f0384234640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f037c0facd0 0x7f037c111870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:24.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f0384234640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f037c0fb680 0x7f037c111db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:24.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f0384234640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f037c102780 0x7f037c10cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:24.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f0384234640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c10d0e0 con 0x7f037c0fb680 2024-09-10T19:19:24.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f0384234640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037c10d250 con 0x7f037c0facd0 2024-09-10T19:19:24.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f0384234640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f037c10d410 con 0x7f037c102780 2024-09-10T19:19:24.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f03817cf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f037c0fb680 0x7f037c111db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:24.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f03827d1640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f037c102780 0x7f037c10cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:24.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f03817cf640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f037c0fb680 0x7f037c111db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53628/0 (socket says 172.21.15.119:53628) 2024-09-10T19:19:24.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f03817cf640 1 -- 172.21.15.119:0/2338788747 learned_addr learned my addr 172.21.15.119:0/2338788747 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:19:24.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f0381fd0640 1 --2- 172.21.15.119:0/2338788747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f037c0facd0 0x7f037c111870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:24.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f03827d1640 1 -- 172.21.15.119:0/2338788747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f037c0facd0 msgr2=0x7f037c111870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:24.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f03827d1640 1 --2- 172.21.15.119:0/2338788747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f037c0facd0 0x7f037c111870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:24.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f03827d1640 1 -- 172.21.15.119:0/2338788747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f037c0fb680 msgr2=0x7f037c111db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:24.452 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f03827d1640 1 --2- 172.21.15.119:0/2338788747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f037c0fb680 0x7f037c111db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:24.452 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f03827d1640 1 -- 172.21.15.119:0/2338788747 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c005040 con 0x7f037c102780 2024-09-10T19:19:24.452 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.446+0000 7f03817cf640 1 --2- 172.21.15.119:0/2338788747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f037c0fb680 0x7f037c111db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:19:24.452 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.447+0000 7f03827d1640 1 --2- 172.21.15.119:0/2338788747 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f037c102780 0x7f037c10cae0 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f037402a4c0 tx=0x7f0374009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:19:24.452 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.447+0000 7f03727fc640 1 -- 172.21.15.119:0/2338788747 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037402bd60 con 0x7f037c102780 2024-09-10T19:19:24.452 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.447+0000 7f03727fc640 1 -- 172.21.15.119:0/2338788747 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f037403c8b0 con 0x7f037c102780 2024-09-10T19:19:24.452 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.447+0000 7f03727fc640 1 -- 172.21.15.119:0/2338788747 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0374033770 con 0x7f037c102780 2024-09-10T19:19:24.452 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.447+0000 7f0384234640 1 -- 172.21.15.119:0/2338788747 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0374005040 con 0x7f037c102780 2024-09-10T19:19:24.452 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.447+0000 7f0384234640 1 -- 172.21.15.119:0/2338788747 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f037c1197a0 con 0x7f037c102780 2024-09-10T19:19:24.452 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.448+0000 7f0384234640 1 -- 172.21.15.119:0/2338788747 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f037c0fdff0 con 0x7f037c102780 2024-09-10T19:19:24.452 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.448+0000 7f03727fc640 1 -- 172.21.15.119:0/2338788747 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f0374030020 con 0x7f037c102780 2024-09-10T19:19:24.452 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.449+0000 7f03727fc640 1 --2- 172.21.15.119:0/2338788747 >> v2:172.21.15.119:6800/3118961733 conn(0x7f036806ef80 0x7f0368071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:24.452 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.449+0000 7f03727fc640 1 -- 172.21.15.119:0/2338788747 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(751..751 src has 1..751) v4 ==== 7855+0+0 (secure 0 0 0) 0x7f03740ae6e0 con 0x7f037c102780 2024-09-10T19:19:24.453 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.449+0000 7f0381fd0640 1 --2- 172.21.15.119:0/2338788747 >> v2:172.21.15.119:6800/3118961733 conn(0x7f036806ef80 0x7f0368071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:24.453 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.449+0000 7f03727fc640 1 -- 172.21.15.119:0/2338788747 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=752}) v3 -- 0x7f036807a3c0 con 0x7f037c102780 2024-09-10T19:19:24.453 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.449+0000 7f0381fd0640 1 --2- 172.21.15.119:0/2338788747 >> v2:172.21.15.119:6800/3118961733 conn(0x7f036806ef80 0x7f0368071430 secure :-1 s=READY pgs=3723 cs=0 l=1 rev1=1 crypto rx=0x7f0378001250 tx=0x7f0378004d00 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:19:24.454 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.453+0000 7f03727fc640 1 -- 172.21.15.119:0/2338788747 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f037407b010 con 0x7f037c102780 2024-09-10T19:19:24.608 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.607+0000 7f0384234640 1 -- 172.21.15.119:0/2338788747 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f037c10df90 con 0x7f037c102780 2024-09-10T19:19:24.696 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.694+0000 7f03727fc640 1 -- 172.21.15.119:0/2338788747 <== mon.2 v2:172.21.15.119:3301/0 7 ==== osd_map(752..752 src has 1..752) v4 ==== 620+0+0 (secure 0 0 0) 0x7f037407ad50 con 0x7f037c102780 2024-09-10T19:19:24.696 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.694+0000 7f03727fc640 1 -- 172.21.15.119:0/2338788747 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=753}) v3 -- 0x7f036807af90 con 0x7f037c102780 2024-09-10T19:19:24.701 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.699+0000 7f03727fc640 1 -- 172.21.15.119:0/2338788747 <== mon.2 v2:172.21.15.119:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 791ac3f1-f414-442d-8950-49875e1f19bf v752) v1 ==== 217+0+0 (secure 0 0 0) 0x7f037407a9b0 con 0x7f037c102780 2024-09-10T19:19:24.796 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:24.795+0000 7f0384234640 1 -- 172.21.15.119:0/2338788747 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f037c04bf60 con 0x7f037c102780 2024-09-10T19:19:25.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:24 smithi194 ceph-mon[32464]: pgmap v1206: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:25.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:24 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2338788747' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:19:25.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:24 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:19:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:24 smithi119 ceph-mon[29577]: pgmap v1206: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:24 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2338788747' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:19:25.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:24 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:19:25.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:24 smithi119 ceph-mon[40674]: pgmap v1206: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:25.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:24 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2338788747' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:19:25.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:24 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:19:25.700 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.698+0000 7f03727fc640 1 -- 172.21.15.119:0/2338788747 <== mon.2 v2:172.21.15.119:3301/0 9 ==== osd_map(753..753 src has 1..753) v4 ==== 620+0+0 (secure 0 0 0) 0x7f037407b940 con 0x7f037c102780 2024-09-10T19:19:25.700 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.698+0000 7f03727fc640 1 -- 172.21.15.119:0/2338788747 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=754}) v3 -- 0x7f036807b420 con 0x7f037c102780 2024-09-10T19:19:25.705 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.703+0000 7f03727fc640 1 -- 172.21.15.119:0/2338788747 <== mon.2 v2:172.21.15.119:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 791ac3f1-f414-442d-8950-49875e1f19bf v753) v1 ==== 217+0+0 (secure 0 0 0) 0x7f037407ab90 con 0x7f037c102780 2024-09-10T19:19:25.705 INFO:tasks.workunit.client.0.smithi119.stderr:set-quota max_bytes = 0 for pool 791ac3f1-f414-442d-8950-49875e1f19bf 2024-09-10T19:19:25.708 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.706+0000 7f0384234640 1 -- 172.21.15.119:0/2338788747 >> v2:172.21.15.119:6800/3118961733 conn(0x7f036806ef80 msgr2=0x7f0368071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:25.708 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.706+0000 7f0384234640 1 --2- 172.21.15.119:0/2338788747 >> v2:172.21.15.119:6800/3118961733 conn(0x7f036806ef80 0x7f0368071430 secure :-1 s=READY pgs=3723 cs=0 l=1 rev1=1 crypto rx=0x7f0378001250 tx=0x7f0378004d00 comp rx=0 tx=0).stop 2024-09-10T19:19:25.708 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.706+0000 7f0384234640 1 -- 172.21.15.119:0/2338788747 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f037c102780 msgr2=0x7f037c10cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:25.708 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.706+0000 7f0384234640 1 --2- 172.21.15.119:0/2338788747 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f037c102780 0x7f037c10cae0 secure :-1 s=READY pgs=2950 cs=0 l=1 rev1=1 crypto rx=0x7f037402a4c0 tx=0x7f0374009da0 comp rx=0 tx=0).stop 2024-09-10T19:19:25.708 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.706+0000 7f0384234640 1 -- 172.21.15.119:0/2338788747 shutdown_connections 2024-09-10T19:19:25.708 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.706+0000 7f0384234640 1 --2- 172.21.15.119:0/2338788747 >> v2:172.21.15.119:6800/3118961733 conn(0x7f036806ef80 0x7f0368071430 unknown :-1 s=CLOSED pgs=3723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:25.709 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.706+0000 7f0384234640 1 --2- 172.21.15.119:0/2338788747 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f037c102780 0x7f037c10cae0 unknown :-1 s=CLOSED pgs=2950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:25.709 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.706+0000 7f0384234640 1 --2- 172.21.15.119:0/2338788747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f037c0fb680 0x7f037c111db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:25.709 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.706+0000 7f0384234640 1 --2- 172.21.15.119:0/2338788747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f037c0facd0 0x7f037c111870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:25.709 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.706+0000 7f0384234640 1 -- 172.21.15.119:0/2338788747 >> 172.21.15.119:0/2338788747 conn(0x7f037c074b80 msgr2=0x7f037c071840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:19:25.709 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.706+0000 7f0384234640 1 -- 172.21.15.119:0/2338788747 shutdown_connections 2024-09-10T19:19:25.709 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.706+0000 7f0384234640 1 -- 172.21.15.119:0/2338788747 wait complete. 2024-09-10T19:19:25.717 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd pool set-quota 791ac3f1-f414-442d-8950-49875e1f19bf max_objects 0 2024-09-10T19:19:25.799 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.795+0000 7f9a6f19d640 1 -- 172.21.15.119:0/854834284 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9a68101720 msgr2=0x7f9a68101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:25.799 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.795+0000 7f9a6f19d640 1 --2- 172.21.15.119:0/854834284 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9a68101720 0x7f9a68101af0 secure :-1 s=READY pgs=2951 cs=0 l=1 rev1=1 crypto rx=0x7f9a64004890 tx=0x7f9a64009da0 comp rx=0 tx=0).stop 2024-09-10T19:19:25.799 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.795+0000 7f9a6f19d640 1 -- 172.21.15.119:0/854834284 shutdown_connections 2024-09-10T19:19:25.799 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.795+0000 7f9a6f19d640 1 --2- 172.21.15.119:0/854834284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a680fbf90 0x7f9a68108520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:25.799 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.795+0000 7f9a6f19d640 1 --2- 172.21.15.119:0/854834284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a68102030 0x7f9a680fb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:25.799 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.795+0000 7f9a6f19d640 1 --2- 172.21.15.119:0/854834284 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9a68101720 0x7f9a68101af0 unknown :-1 s=CLOSED pgs=2951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:25.799 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.795+0000 7f9a6f19d640 1 -- 172.21.15.119:0/854834284 >> 172.21.15.119:0/854834284 conn(0x7f9a68072600 msgr2=0x7f9a68072a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:19:25.800 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.795+0000 7f9a6f19d640 1 -- 172.21.15.119:0/854834284 shutdown_connections 2024-09-10T19:19:25.800 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.795+0000 7f9a6f19d640 1 -- 172.21.15.119:0/854834284 wait complete. 2024-09-10T19:19:25.800 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.795+0000 7f9a6f19d640 1 Processor -- start 2024-09-10T19:19:25.800 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.795+0000 7f9a6f19d640 1 -- start start 2024-09-10T19:19:25.800 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6f19d640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a680fbf90 0x7f9a68112ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:25.800 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6f19d640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a68101720 0x7f9a68113220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:25.800 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6f19d640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9a68102030 0x7f9a6810ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:25.800 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6f19d640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6810d4c0 con 0x7f9a680fbf90 2024-09-10T19:19:25.800 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6f19d640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6810d630 con 0x7f9a68101720 2024-09-10T19:19:25.800 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6f19d640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6810d7f0 con 0x7f9a68102030 2024-09-10T19:19:25.800 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6cf39640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a680fbf90 0x7f9a68112ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:25.801 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6d73a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9a68102030 0x7f9a6810ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:25.801 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6cf39640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a680fbf90 0x7f9a68112ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53642/0 (socket says 172.21.15.119:53642) 2024-09-10T19:19:25.801 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6d73a640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9a68102030 0x7f9a6810ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:44036/0 (socket says 172.21.15.119:44036) 2024-09-10T19:19:25.801 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6cf39640 1 -- 172.21.15.119:0/294172086 learned_addr learned my addr 172.21.15.119:0/294172086 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:19:25.801 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a5ffff640 1 --2- 172.21.15.119:0/294172086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a68101720 0x7f9a68113220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:25.801 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6cf39640 1 -- 172.21.15.119:0/294172086 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9a68102030 msgr2=0x7f9a6810ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:25.801 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6cf39640 1 --2- 172.21.15.119:0/294172086 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9a68102030 0x7f9a6810ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:25.801 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6cf39640 1 -- 172.21.15.119:0/294172086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a68101720 msgr2=0x7f9a68113220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:25.801 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6cf39640 1 --2- 172.21.15.119:0/294172086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a68101720 0x7f9a68113220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:25.801 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6cf39640 1 -- 172.21.15.119:0/294172086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a600098b0 con 0x7f9a680fbf90 2024-09-10T19:19:25.801 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6d73a640 1 --2- 172.21.15.119:0/294172086 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9a68102030 0x7f9a6810ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:19:25.801 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.796+0000 7f9a6cf39640 1 --2- 172.21.15.119:0/294172086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a680fbf90 0x7f9a68112ce0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f9a6402d040 tx=0x7f9a64035040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:19:25.802 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.797+0000 7f9a5d7fa640 1 -- 172.21.15.119:0/294172086 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a640093e0 con 0x7f9a680fbf90 2024-09-10T19:19:25.802 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.797+0000 7f9a5d7fa640 1 -- 172.21.15.119:0/294172086 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f9a64009540 con 0x7f9a680fbf90 2024-09-10T19:19:25.802 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.797+0000 7f9a6f19d640 1 -- 172.21.15.119:0/294172086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a64005040 con 0x7f9a680fbf90 2024-09-10T19:19:25.802 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.797+0000 7f9a5d7fa640 1 -- 172.21.15.119:0/294172086 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6403c680 con 0x7f9a680fbf90 2024-09-10T19:19:25.802 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.797+0000 7f9a6f19d640 1 -- 172.21.15.119:0/294172086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a68119b30 con 0x7f9a680fbf90 2024-09-10T19:19:25.803 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.798+0000 7f9a6f19d640 1 -- 172.21.15.119:0/294172086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6804bf60 con 0x7f9a680fbf90 2024-09-10T19:19:25.803 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.798+0000 7f9a5d7fa640 1 -- 172.21.15.119:0/294172086 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f9a64030020 con 0x7f9a680fbf90 2024-09-10T19:19:25.803 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.799+0000 7f9a5d7fa640 1 --2- 172.21.15.119:0/294172086 >> v2:172.21.15.119:6800/3118961733 conn(0x7f9a5006ef80 0x7f9a50071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:25.803 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.799+0000 7f9a5d7fa640 1 -- 172.21.15.119:0/294172086 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(753..753 src has 1..753) v4 ==== 7855+0+0 (secure 0 0 0) 0x7f9a640ae7f0 con 0x7f9a680fbf90 2024-09-10T19:19:25.803 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.799+0000 7f9a5ffff640 1 --2- 172.21.15.119:0/294172086 >> v2:172.21.15.119:6800/3118961733 conn(0x7f9a5006ef80 0x7f9a50071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:25.803 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.799+0000 7f9a5d7fa640 1 -- 172.21.15.119:0/294172086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=754}) v3 -- 0x7f9a5007a340 con 0x7f9a680fbf90 2024-09-10T19:19:25.804 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.799+0000 7f9a5ffff640 1 --2- 172.21.15.119:0/294172086 >> v2:172.21.15.119:6800/3118961733 conn(0x7f9a5006ef80 0x7f9a50071430 secure :-1 s=READY pgs=3724 cs=0 l=1 rev1=1 crypto rx=0x7f9a6810e5a0 tx=0x7f9a5800b040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:19:25.804 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.803+0000 7f9a5d7fa640 1 -- 172.21.15.119:0/294172086 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f9a6407b1b0 con 0x7f9a680fbf90 2024-09-10T19:19:25.961 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:25.959+0000 7f9a6f19d640 1 -- 172.21.15.119:0/294172086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f9a6810e3e0 con 0x7f9a680fbf90 2024-09-10T19:19:26.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]': finished 2024-09-10T19:19:26.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:25 smithi194 ceph-mon[32464]: osdmap e752: 8 total, 8 up, 8 in 2024-09-10T19:19:26.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:25 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/2338788747' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:19:26.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:25 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:19:26.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]': finished 2024-09-10T19:19:26.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:25 smithi119 ceph-mon[29577]: osdmap e752: 8 total, 8 up, 8 in 2024-09-10T19:19:26.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:25 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/2338788747' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:19:26.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:25 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:19:26.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]': finished 2024-09-10T19:19:26.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:25 smithi119 ceph-mon[40674]: osdmap e752: 8 total, 8 up, 8 in 2024-09-10T19:19:26.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:25 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/2338788747' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:19:26.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:25 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:19:26.701 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:26.699+0000 7f9a5d7fa640 1 -- 172.21.15.119:0/294172086 <== mon.0 v2:172.21.15.119:3300/0 7 ==== osd_map(754..754 src has 1..754) v4 ==== 620+0+0 (secure 0 0 0) 0x7f9a6407aef0 con 0x7f9a680fbf90 2024-09-10T19:19:26.701 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:26.699+0000 7f9a5d7fa640 1 -- 172.21.15.119:0/294172086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=755}) v3 -- 0x7f9a5007af10 con 0x7f9a680fbf90 2024-09-10T19:19:26.703 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:26.702+0000 7f9a5d7fa640 1 -- 172.21.15.119:0/294172086 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 791ac3f1-f414-442d-8950-49875e1f19bf v754) v1 ==== 221+0+0 (secure 0 0 0) 0x7f9a6407ab50 con 0x7f9a680fbf90 2024-09-10T19:19:26.802 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:26.801+0000 7f9a6f19d640 1 -- 172.21.15.119:0/294172086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7f9a680fb630 con 0x7f9a680fbf90 2024-09-10T19:19:27.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:26 smithi194 ceph-mon[32464]: pgmap v1208: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:19:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:26 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]': finished 2024-09-10T19:19:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:26 smithi194 ceph-mon[32464]: osdmap e753: 8 total, 8 up, 8 in 2024-09-10T19:19:27.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:26 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/294172086' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:19:27.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:26 smithi119 ceph-mon[29577]: pgmap v1208: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:19:27.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:26 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]': finished 2024-09-10T19:19:27.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:26 smithi119 ceph-mon[29577]: osdmap e753: 8 total, 8 up, 8 in 2024-09-10T19:19:27.067 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:26 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/294172086' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:19:27.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:26 smithi119 ceph-mon[40674]: pgmap v1208: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:19:27.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:26 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]': finished 2024-09-10T19:19:27.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:26 smithi119 ceph-mon[40674]: osdmap e753: 8 total, 8 up, 8 in 2024-09-10T19:19:27.067 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:26 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/294172086' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:19:27.341 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.339+0000 7f9a5d7fa640 1 -- 172.21.15.119:0/294172086 <== mon.0 v2:172.21.15.119:3300/0 9 ==== osd_map(755..755 src has 1..755) v4 ==== 620+0+0 (secure 0 0 0) 0x7f9a6407e580 con 0x7f9a680fbf90 2024-09-10T19:19:27.341 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.339+0000 7f9a5d7fa640 1 -- 172.21.15.119:0/294172086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=756}) v3 -- 0x7f9a5007b3e0 con 0x7f9a680fbf90 2024-09-10T19:19:27.344 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.342+0000 7f9a5d7fa640 1 -- 172.21.15.119:0/294172086 <== mon.0 v2:172.21.15.119:3300/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 791ac3f1-f414-442d-8950-49875e1f19bf v755) v1 ==== 221+0+0 (secure 0 0 0) 0x7f9a6407ad30 con 0x7f9a680fbf90 2024-09-10T19:19:27.344 INFO:tasks.workunit.client.0.smithi119.stderr:set-quota max_objects = 0 for pool 791ac3f1-f414-442d-8950-49875e1f19bf 2024-09-10T19:19:27.348 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.346+0000 7f9a6f19d640 1 -- 172.21.15.119:0/294172086 >> v2:172.21.15.119:6800/3118961733 conn(0x7f9a5006ef80 msgr2=0x7f9a50071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:27.348 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.346+0000 7f9a6f19d640 1 --2- 172.21.15.119:0/294172086 >> v2:172.21.15.119:6800/3118961733 conn(0x7f9a5006ef80 0x7f9a50071430 secure :-1 s=READY pgs=3724 cs=0 l=1 rev1=1 crypto rx=0x7f9a6810e5a0 tx=0x7f9a5800b040 comp rx=0 tx=0).stop 2024-09-10T19:19:27.348 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.346+0000 7f9a6f19d640 1 -- 172.21.15.119:0/294172086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a680fbf90 msgr2=0x7f9a68112ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:27.348 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.346+0000 7f9a6f19d640 1 --2- 172.21.15.119:0/294172086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a680fbf90 0x7f9a68112ce0 secure :-1 s=READY pgs=2452 cs=0 l=1 rev1=1 crypto rx=0x7f9a6402d040 tx=0x7f9a64035040 comp rx=0 tx=0).stop 2024-09-10T19:19:27.348 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.346+0000 7f9a6f19d640 1 -- 172.21.15.119:0/294172086 shutdown_connections 2024-09-10T19:19:27.349 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.346+0000 7f9a6f19d640 1 --2- 172.21.15.119:0/294172086 >> v2:172.21.15.119:6800/3118961733 conn(0x7f9a5006ef80 0x7f9a50071430 unknown :-1 s=CLOSED pgs=3724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:27.349 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.346+0000 7f9a6f19d640 1 --2- 172.21.15.119:0/294172086 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9a68102030 0x7f9a6810ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:27.349 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.346+0000 7f9a6f19d640 1 --2- 172.21.15.119:0/294172086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a68101720 0x7f9a68113220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:27.349 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.346+0000 7f9a6f19d640 1 --2- 172.21.15.119:0/294172086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a680fbf90 0x7f9a68112ce0 unknown :-1 s=CLOSED pgs=2452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:27.349 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.346+0000 7f9a6f19d640 1 -- 172.21.15.119:0/294172086 >> 172.21.15.119:0/294172086 conn(0x7f9a68072600 msgr2=0x7f9a68100510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:19:27.349 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.346+0000 7f9a6f19d640 1 -- 172.21.15.119:0/294172086 shutdown_connections 2024-09-10T19:19:27.349 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.346+0000 7f9a6f19d640 1 -- 172.21.15.119:0/294172086 wait complete. 2024-09-10T19:19:27.357 INFO:tasks.workunit.client.0.smithi119.stderr:+ wait 148713 2024-09-10T19:19:27.357 INFO:tasks.workunit.client.0.smithi119.stderr:+ '[' 0 -ne 0 ']' 2024-09-10T19:19:27.358 INFO:tasks.workunit.client.0.smithi119.stderr:+ true 2024-09-10T19:19:27.358 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put three /etc/passwd 2024-09-10T19:19:27.388 INFO:tasks.workunit.client.0.smithi119.stderr:++ uuidgen 2024-09-10T19:19:27.389 INFO:tasks.workunit.client.0.smithi119.stderr:+ pp=0ceff89d-376b-4634-ba36-dca6344065fb 2024-09-10T19:19:27.389 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd pool create 0ceff89d-376b-4634-ba36-dca6344065fb 12 2024-09-10T19:19:27.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.463+0000 7fed4060f640 1 -- 172.21.15.119:0/4203039066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed381015f0 msgr2=0x7fed381019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:27.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.463+0000 7fed4060f640 1 --2- 172.21.15.119:0/4203039066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed381015f0 0x7fed381019c0 secure :-1 s=READY pgs=2454 cs=0 l=1 rev1=1 crypto rx=0x7fed2c004950 tx=0x7fed2c009da0 comp rx=0 tx=0).stop 2024-09-10T19:19:27.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.463+0000 7fed4060f640 1 -- 172.21.15.119:0/4203039066 shutdown_connections 2024-09-10T19:19:27.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.463+0000 7fed4060f640 1 --2- 172.21.15.119:0/4203039066 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed380fbd30 0x7fed381083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:27.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.463+0000 7fed4060f640 1 --2- 172.21.15.119:0/4203039066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed38101f90 0x7fed380fb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:27.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.463+0000 7fed4060f640 1 --2- 172.21.15.119:0/4203039066 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed381015f0 0x7fed381019c0 unknown :-1 s=CLOSED pgs=2454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:27.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.463+0000 7fed4060f640 1 -- 172.21.15.119:0/4203039066 >> 172.21.15.119:0/4203039066 conn(0x7fed380725f0 msgr2=0x7fed380f9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:19:27.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.463+0000 7fed4060f640 1 -- 172.21.15.119:0/4203039066 shutdown_connections 2024-09-10T19:19:27.467 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.463+0000 7fed4060f640 1 -- 172.21.15.119:0/4203039066 wait complete. 2024-09-10T19:19:27.468 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.463+0000 7fed4060f640 1 Processor -- start 2024-09-10T19:19:27.468 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.463+0000 7fed4060f640 1 -- start start 2024-09-10T19:19:27.468 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed4060f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed380fbd30 0x7fed3810dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:27.468 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed4060f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed381015f0 0x7fed3810e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:27.468 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed4060f640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed38101f90 0x7fed3810e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:27.468 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed4060f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed38118fb0 con 0x7fed380fbd30 2024-09-10T19:19:27.468 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed4060f640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed38119120 con 0x7fed381015f0 2024-09-10T19:19:27.468 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed4060f640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed381192e0 con 0x7fed38101f90 2024-09-10T19:19:27.468 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed3e3ab640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed380fbd30 0x7fed3810dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:27.468 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed3e3ab640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed380fbd30 0x7fed3810dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53714/0 (socket says 172.21.15.119:53714) 2024-09-10T19:19:27.468 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed3e3ab640 1 -- 172.21.15.119:0/3379186574 learned_addr learned my addr 172.21.15.119:0/3379186574 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:19:27.468 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed3ebac640 1 --2- 172.21.15.119:0/3379186574 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed38101f90 0x7fed3810e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:27.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed3dbaa640 1 --2- 172.21.15.119:0/3379186574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed381015f0 0x7fed3810e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:27.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed3e3ab640 1 -- 172.21.15.119:0/3379186574 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed38101f90 msgr2=0x7fed3810e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:27.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed3e3ab640 1 --2- 172.21.15.119:0/3379186574 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed38101f90 0x7fed3810e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:27.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed3e3ab640 1 -- 172.21.15.119:0/3379186574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed381015f0 msgr2=0x7fed3810e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:27.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed3e3ab640 1 --2- 172.21.15.119:0/3379186574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed381015f0 0x7fed3810e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:27.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed3e3ab640 1 -- 172.21.15.119:0/3379186574 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed300098b0 con 0x7fed380fbd30 2024-09-10T19:19:27.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed3dbaa640 1 --2- 172.21.15.119:0/3379186574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed381015f0 0x7fed3810e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T19:19:27.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed3e3ab640 1 --2- 172.21.15.119:0/3379186574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed380fbd30 0x7fed3810dd80 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fed2c02d040 tx=0x7fed2c035040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:19:27.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed2affd640 1 -- 172.21.15.119:0/3379186574 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c009850 con 0x7fed380fbd30 2024-09-10T19:19:27.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed2affd640 1 -- 172.21.15.119:0/3379186574 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fed2c02ab70 con 0x7fed380fbd30 2024-09-10T19:19:27.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed4060f640 1 -- 172.21.15.119:0/3379186574 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed2c005040 con 0x7fed380fbd30 2024-09-10T19:19:27.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed2affd640 1 -- 172.21.15.119:0/3379186574 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed2c033420 con 0x7fed380fbd30 2024-09-10T19:19:27.469 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.464+0000 7fed4060f640 1 -- 172.21.15.119:0/3379186574 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed38119970 con 0x7fed380fbd30 2024-09-10T19:19:27.470 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.465+0000 7fed4060f640 1 -- 172.21.15.119:0/3379186574 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed380fcd70 con 0x7fed380fbd30 2024-09-10T19:19:27.470 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.466+0000 7fed2affd640 1 -- 172.21.15.119:0/3379186574 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7fed2c030020 con 0x7fed380fbd30 2024-09-10T19:19:27.470 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.466+0000 7fed2affd640 1 --2- 172.21.15.119:0/3379186574 >> v2:172.21.15.119:6800/3118961733 conn(0x7fed2406ef80 0x7fed24071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:27.470 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.466+0000 7fed2affd640 1 -- 172.21.15.119:0/3379186574 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(755..755 src has 1..755) v4 ==== 7855+0+0 (secure 0 0 0) 0x7fed2c0aedb0 con 0x7fed380fbd30 2024-09-10T19:19:27.471 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.466+0000 7fed3dbaa640 1 --2- 172.21.15.119:0/3379186574 >> v2:172.21.15.119:6800/3118961733 conn(0x7fed2406ef80 0x7fed24071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:27.471 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.467+0000 7fed3dbaa640 1 --2- 172.21.15.119:0/3379186574 >> v2:172.21.15.119:6800/3118961733 conn(0x7fed2406ef80 0x7fed24071430 secure :-1 s=READY pgs=3726 cs=0 l=1 rev1=1 crypto rx=0x7fed380664a0 tx=0x7fed34009040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:19:27.472 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.471+0000 7fed2affd640 1 -- 172.21.15.119:0/3379186574 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fed2c07b810 con 0x7fed380fbd30 2024-09-10T19:19:27.629 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:27.628+0000 7fed4060f640 1 -- 172.21.15.119:0/3379186574 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pg_num": 12} v 0) v1 -- 0x7fed3804bf60 con 0x7fed380fbd30 2024-09-10T19:19:28.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/294172086' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]': finished 2024-09-10T19:19:28.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:27 smithi194 ceph-mon[32464]: osdmap e754: 8 total, 8 up, 8 in 2024-09-10T19:19:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/294172086' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:19:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:27 smithi194 ceph-mon[32464]: pool '791ac3f1-f414-442d-8950-49875e1f19bf' no longer out of quota; removing NO_QUOTA flag 2024-09-10T19:19:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:27 smithi194 ceph-mon[32464]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T19:19:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:27 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/294172086' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]': finished 2024-09-10T19:19:28.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:27 smithi194 ceph-mon[32464]: osdmap e755: 8 total, 8 up, 8 in 2024-09-10T19:19:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/294172086' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]': finished 2024-09-10T19:19:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:27 smithi119 ceph-mon[29577]: osdmap e754: 8 total, 8 up, 8 in 2024-09-10T19:19:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/294172086' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:19:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:27 smithi119 ceph-mon[29577]: pool '791ac3f1-f414-442d-8950-49875e1f19bf' no longer out of quota; removing NO_QUOTA flag 2024-09-10T19:19:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:27 smithi119 ceph-mon[29577]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T19:19:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:27 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/294172086' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]': finished 2024-09-10T19:19:28.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:27 smithi119 ceph-mon[29577]: osdmap e755: 8 total, 8 up, 8 in 2024-09-10T19:19:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/294172086' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]': finished 2024-09-10T19:19:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:27 smithi119 ceph-mon[40674]: osdmap e754: 8 total, 8 up, 8 in 2024-09-10T19:19:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/294172086' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:19:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:27 smithi119 ceph-mon[40674]: pool '791ac3f1-f414-442d-8950-49875e1f19bf' no longer out of quota; removing NO_QUOTA flag 2024-09-10T19:19:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:27 smithi119 ceph-mon[40674]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T19:19:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:27 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/294172086' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]': finished 2024-09-10T19:19:28.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:27 smithi119 ceph-mon[40674]: osdmap e755: 8 total, 8 up, 8 in 2024-09-10T19:19:28.350 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.348+0000 7fed2affd640 1 -- 172.21.15.119:0/3379186574 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pg_num": 12}]=0 pool '0ceff89d-376b-4634-ba36-dca6344065fb' created v756) v1 ==== 176+0+0 (secure 0 0 0) 0x7fed2c07b1b0 con 0x7fed380fbd30 2024-09-10T19:19:28.447 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.446+0000 7fed4060f640 1 -- 172.21.15.119:0/3379186574 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pg_num": 12} v 0) v1 -- 0x7fed38066250 con 0x7fed380fbd30 2024-09-10T19:19:28.448 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.446+0000 7fed2affd640 1 -- 172.21.15.119:0/3379186574 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pg_num": 12}]=0 pool '0ceff89d-376b-4634-ba36-dca6344065fb' already exists v756) v1 ==== 183+0+0 (secure 0 0 0) 0x7fed2c07b390 con 0x7fed380fbd30 2024-09-10T19:19:28.448 INFO:tasks.workunit.client.0.smithi119.stderr:pool '0ceff89d-376b-4634-ba36-dca6344065fb' already exists 2024-09-10T19:19:28.450 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.448+0000 7fed4060f640 1 -- 172.21.15.119:0/3379186574 >> v2:172.21.15.119:6800/3118961733 conn(0x7fed2406ef80 msgr2=0x7fed24071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:28.450 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.448+0000 7fed4060f640 1 --2- 172.21.15.119:0/3379186574 >> v2:172.21.15.119:6800/3118961733 conn(0x7fed2406ef80 0x7fed24071430 secure :-1 s=READY pgs=3726 cs=0 l=1 rev1=1 crypto rx=0x7fed380664a0 tx=0x7fed34009040 comp rx=0 tx=0).stop 2024-09-10T19:19:28.450 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.448+0000 7fed4060f640 1 -- 172.21.15.119:0/3379186574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed380fbd30 msgr2=0x7fed3810dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:28.450 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.448+0000 7fed4060f640 1 --2- 172.21.15.119:0/3379186574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed380fbd30 0x7fed3810dd80 secure :-1 s=READY pgs=2455 cs=0 l=1 rev1=1 crypto rx=0x7fed2c02d040 tx=0x7fed2c035040 comp rx=0 tx=0).stop 2024-09-10T19:19:28.450 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.448+0000 7fed4060f640 1 -- 172.21.15.119:0/3379186574 shutdown_connections 2024-09-10T19:19:28.450 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.448+0000 7fed4060f640 1 --2- 172.21.15.119:0/3379186574 >> v2:172.21.15.119:6800/3118961733 conn(0x7fed2406ef80 0x7fed24071430 unknown :-1 s=CLOSED pgs=3726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:28.450 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.448+0000 7fed4060f640 1 --2- 172.21.15.119:0/3379186574 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fed38101f90 0x7fed3810e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:28.450 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.448+0000 7fed4060f640 1 --2- 172.21.15.119:0/3379186574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed381015f0 0x7fed3810e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:28.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.448+0000 7fed4060f640 1 --2- 172.21.15.119:0/3379186574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed380fbd30 0x7fed3810dd80 unknown :-1 s=CLOSED pgs=2455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:28.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.448+0000 7fed4060f640 1 -- 172.21.15.119:0/3379186574 >> 172.21.15.119:0/3379186574 conn(0x7fed380725f0 msgr2=0x7fed380ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:19:28.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.448+0000 7fed4060f640 1 -- 172.21.15.119:0/3379186574 shutdown_connections 2024-09-10T19:19:28.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.448+0000 7fed4060f640 1 -- 172.21.15.119:0/3379186574 wait complete. 2024-09-10T19:19:28.459 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd pool application enable 0ceff89d-376b-4634-ba36-dca6344065fb rados 2024-09-10T19:19:28.537 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.533+0000 7f9926b54640 1 -- 172.21.15.119:0/3904901199 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99201030d0 msgr2=0x7f99200fac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:28.537 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.533+0000 7f9926b54640 1 --2- 172.21.15.119:0/3904901199 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99201030d0 0x7f99200fac30 secure :-1 s=READY pgs=2954 cs=0 l=1 rev1=1 crypto rx=0x7f990c004950 tx=0x7f990c009da0 comp rx=0 tx=0).stop 2024-09-10T19:19:28.537 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.533+0000 7f9926b54640 1 -- 172.21.15.119:0/3904901199 shutdown_connections 2024-09-10T19:19:28.537 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.533+0000 7f9926b54640 1 --2- 172.21.15.119:0/3904901199 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99200fb310 0x7f9920109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:28.537 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.533+0000 7f9926b54640 1 --2- 172.21.15.119:0/3904901199 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99201030d0 0x7f99200fac30 unknown :-1 s=CLOSED pgs=2954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:28.537 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.533+0000 7f9926b54640 1 --2- 172.21.15.119:0/3904901199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99201027c0 0x7f9920102b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:28.537 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.533+0000 7f9926b54640 1 -- 172.21.15.119:0/3904901199 >> 172.21.15.119:0/3904901199 conn(0x7f9920074b80 msgr2=0x7f99200f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:19:28.537 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.533+0000 7f9926b54640 1 -- 172.21.15.119:0/3904901199 shutdown_connections 2024-09-10T19:19:28.537 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.533+0000 7f9926b54640 1 -- 172.21.15.119:0/3904901199 wait complete. 2024-09-10T19:19:28.537 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.533+0000 7f9926b54640 1 Processor -- start 2024-09-10T19:19:28.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f9926b54640 1 -- start start 2024-09-10T19:19:28.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f9926b54640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99200fb310 0x7f9920112c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:28.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f9926b54640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99201027c0 0x7f99201131c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:28.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f9926b54640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99201030d0 0x7f992010ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:28.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f9926b54640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992010d4a0 con 0x7f99200fb310 2024-09-10T19:19:28.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f9926b54640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992010d610 con 0x7f99201030d0 2024-09-10T19:19:28.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f9926b54640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f992010d7d0 con 0x7f99201027c0 2024-09-10T19:19:28.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f99248f0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99200fb310 0x7f9920112c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:28.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f991ffff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99201027c0 0x7f99201131c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:28.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f991ffff640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99201027c0 0x7f99201131c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:44084/0 (socket says 172.21.15.119:44084) 2024-09-10T19:19:28.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f99248f0640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99200fb310 0x7f9920112c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53732/0 (socket says 172.21.15.119:53732) 2024-09-10T19:19:28.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f991ffff640 1 -- 172.21.15.119:0/1165874660 learned_addr learned my addr 172.21.15.119:0/1165874660 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:19:28.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f991ffff640 1 -- 172.21.15.119:0/1165874660 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99201030d0 msgr2=0x7f992010ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:28.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f991ffff640 1 --2- 172.21.15.119:0/1165874660 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99201030d0 0x7f992010ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:28.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f991ffff640 1 -- 172.21.15.119:0/1165874660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99200fb310 msgr2=0x7f9920112c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:28.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f991ffff640 1 --2- 172.21.15.119:0/1165874660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99200fb310 0x7f9920112c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:28.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f991ffff640 1 -- 172.21.15.119:0/1165874660 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99180098b0 con 0x7f99201027c0 2024-09-10T19:19:28.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f99248f0640 1 --2- 172.21.15.119:0/1165874660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99200fb310 0x7f9920112c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:19:28.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f991ffff640 1 --2- 172.21.15.119:0/1165874660 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99201027c0 0x7f99201131c0 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7f990c009200 tx=0x7f990c009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:19:28.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.534+0000 7f991d7fa640 1 -- 172.21.15.119:0/1165874660 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c02abf0 con 0x7f99201027c0 2024-09-10T19:19:28.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.535+0000 7f991d7fa640 1 -- 172.21.15.119:0/1165874660 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f990c02ad50 con 0x7f99201027c0 2024-09-10T19:19:28.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.535+0000 7f991d7fa640 1 -- 172.21.15.119:0/1165874660 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c033670 con 0x7f99201027c0 2024-09-10T19:19:28.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.535+0000 7f9926b54640 1 -- 172.21.15.119:0/1165874660 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f990c005040 con 0x7f99201027c0 2024-09-10T19:19:28.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.535+0000 7f9926b54640 1 -- 172.21.15.119:0/1165874660 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9920119af0 con 0x7f99201027c0 2024-09-10T19:19:28.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.535+0000 7f9926b54640 1 -- 172.21.15.119:0/1165874660 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99200fdf50 con 0x7f99201027c0 2024-09-10T19:19:28.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.536+0000 7f991d7fa640 1 -- 172.21.15.119:0/1165874660 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f990c030020 con 0x7f99201027c0 2024-09-10T19:19:28.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.536+0000 7f991d7fa640 1 --2- 172.21.15.119:0/1165874660 >> v2:172.21.15.119:6800/3118961733 conn(0x7f991006ef80 0x7f9910071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:28.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.536+0000 7f99248f0640 1 --2- 172.21.15.119:0/1165874660 >> v2:172.21.15.119:6800/3118961733 conn(0x7f991006ef80 0x7f9910071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:28.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.536+0000 7f991d7fa640 1 -- 172.21.15.119:0/1165874660 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(756..756 src has 1..756) v4 ==== 8230+0+0 (secure 0 0 0) 0x7f990c0ae5b0 con 0x7f99201027c0 2024-09-10T19:19:28.541 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.537+0000 7f99248f0640 1 --2- 172.21.15.119:0/1165874660 >> v2:172.21.15.119:6800/3118961733 conn(0x7f991006ef80 0x7f9910071430 secure :-1 s=READY pgs=3727 cs=0 l=1 rev1=1 crypto rx=0x7f992010e680 tx=0x7f9914009040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:19:28.542 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.540+0000 7f991d7fa640 1 -- 172.21.15.119:0/1165874660 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f990c07ad70 con 0x7f99201027c0 2024-09-10T19:19:28.697 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:28.695+0000 7f9926b54640 1 -- 172.21.15.119:0/1165874660 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"} v 0) v1 -- 0x7f992010e2f0 con 0x7f99201027c0 2024-09-10T19:19:29.269 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:19:28 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:19:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:28 smithi194 ceph-mon[32464]: pgmap v1212: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:19:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3379186574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pg_num": 12}]: dispatch 2024-09-10T19:19:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3379186574' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pg_num": 12}]': finished 2024-09-10T19:19:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:28 smithi194 ceph-mon[32464]: osdmap e756: 8 total, 8 up, 8 in 2024-09-10T19:19:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3379186574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pg_num": 12}]: dispatch 2024-09-10T19:19:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:28 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1165874660' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]: dispatch 2024-09-10T19:19:29.269 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:28 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]: dispatch 2024-09-10T19:19:29.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:28 smithi119 ceph-mon[29577]: pgmap v1212: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:19:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3379186574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pg_num": 12}]: dispatch 2024-09-10T19:19:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3379186574' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pg_num": 12}]': finished 2024-09-10T19:19:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:28 smithi119 ceph-mon[29577]: osdmap e756: 8 total, 8 up, 8 in 2024-09-10T19:19:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3379186574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pg_num": 12}]: dispatch 2024-09-10T19:19:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:28 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1165874660' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]: dispatch 2024-09-10T19:19:29.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:28 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]: dispatch 2024-09-10T19:19:29.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:28 smithi119 ceph-mon[40674]: pgmap v1212: 176 pgs: 176 active+clean; 732 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:19:29.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3379186574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pg_num": 12}]: dispatch 2024-09-10T19:19:29.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3379186574' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pg_num": 12}]': finished 2024-09-10T19:19:29.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:28 smithi119 ceph-mon[40674]: osdmap e756: 8 total, 8 up, 8 in 2024-09-10T19:19:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3379186574' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pg_num": 12}]: dispatch 2024-09-10T19:19:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:28 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1165874660' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]: dispatch 2024-09-10T19:19:29.318 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:28 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]: dispatch 2024-09-10T19:19:29.353 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:29.352+0000 7f991d7fa640 1 -- 172.21.15.119:0/1165874660 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]=0 enabled application 'rados' on pool '0ceff89d-376b-4634-ba36-dca6344065fb' v757) v1 ==== 213+0+0 (secure 0 0 0) 0x7f990c032e80 con 0x7f99201027c0 2024-09-10T19:19:29.454 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:29.452+0000 7f9926b54640 1 -- 172.21.15.119:0/1165874660 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"} v 0) v1 -- 0x7f9920119c80 con 0x7f99201027c0 2024-09-10T19:19:30.359 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.357+0000 7f991d7fa640 1 -- 172.21.15.119:0/1165874660 <== mon.2 v2:172.21.15.119:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]=0 enabled application 'rados' on pool '0ceff89d-376b-4634-ba36-dca6344065fb' v758) v1 ==== 213+0+0 (secure 0 0 0) 0x7f990c041020 con 0x7f99201027c0 2024-09-10T19:19:30.359 INFO:tasks.workunit.client.0.smithi119.stderr:enabled application 'rados' on pool '0ceff89d-376b-4634-ba36-dca6344065fb' 2024-09-10T19:19:30.362 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.360+0000 7f9926b54640 1 -- 172.21.15.119:0/1165874660 >> v2:172.21.15.119:6800/3118961733 conn(0x7f991006ef80 msgr2=0x7f9910071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:30.362 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.360+0000 7f9926b54640 1 --2- 172.21.15.119:0/1165874660 >> v2:172.21.15.119:6800/3118961733 conn(0x7f991006ef80 0x7f9910071430 secure :-1 s=READY pgs=3727 cs=0 l=1 rev1=1 crypto rx=0x7f992010e680 tx=0x7f9914009040 comp rx=0 tx=0).stop 2024-09-10T19:19:30.362 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.360+0000 7f9926b54640 1 -- 172.21.15.119:0/1165874660 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99201027c0 msgr2=0x7f99201131c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:30.362 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.360+0000 7f9926b54640 1 --2- 172.21.15.119:0/1165874660 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99201027c0 0x7f99201131c0 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7f990c009200 tx=0x7f990c009da0 comp rx=0 tx=0).stop 2024-09-10T19:19:30.363 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.360+0000 7f9926b54640 1 -- 172.21.15.119:0/1165874660 shutdown_connections 2024-09-10T19:19:30.363 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.360+0000 7f9926b54640 1 --2- 172.21.15.119:0/1165874660 >> v2:172.21.15.119:6800/3118961733 conn(0x7f991006ef80 0x7f9910071430 unknown :-1 s=CLOSED pgs=3727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:30.363 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.360+0000 7f9926b54640 1 --2- 172.21.15.119:0/1165874660 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99201030d0 0x7f992010ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:30.363 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.360+0000 7f9926b54640 1 --2- 172.21.15.119:0/1165874660 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f99201027c0 0x7f99201131c0 unknown :-1 s=CLOSED pgs=2955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:30.363 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.360+0000 7f9926b54640 1 --2- 172.21.15.119:0/1165874660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99200fb310 0x7f9920112c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:30.363 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.360+0000 7f9926b54640 1 -- 172.21.15.119:0/1165874660 >> 172.21.15.119:0/1165874660 conn(0x7f9920074b80 msgr2=0x7f99201002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:19:30.363 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.361+0000 7f9926b54640 1 -- 172.21.15.119:0/1165874660 shutdown_connections 2024-09-10T19:19:30.363 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.361+0000 7f9926b54640 1 -- 172.21.15.119:0/1165874660 wait complete. 2024-09-10T19:19:30.373 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd pool set-quota 0ceff89d-376b-4634-ba36-dca6344065fb max_objects 10 2024-09-10T19:19:30.459 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.455+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3929411873 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5b68103070 msgr2=0x7f5b680fc740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:30.459 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.455+0000 7f5b6fe99640 1 --2- 172.21.15.119:0/3929411873 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5b68103070 0x7f5b680fc740 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7f5b580048c0 tx=0x7f5b58009da0 comp rx=0 tx=0).stop 2024-09-10T19:19:30.459 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.455+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3929411873 shutdown_connections 2024-09-10T19:19:30.459 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.455+0000 7f5b6fe99640 1 --2- 172.21.15.119:0/3929411873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b680fce70 0x7f5b68109590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:30.460 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.455+0000 7f5b6fe99640 1 --2- 172.21.15.119:0/3929411873 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5b68103070 0x7f5b680fc740 unknown :-1 s=CLOSED pgs=2956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:30.460 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.455+0000 7f5b6fe99640 1 --2- 172.21.15.119:0/3929411873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b681026d0 0x7f5b68102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:30.460 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.455+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3929411873 >> 172.21.15.119:0/3929411873 conn(0x7f5b680f82e0 msgr2=0x7f5b680fa6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:19:30.460 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.455+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3929411873 shutdown_connections 2024-09-10T19:19:30.460 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.455+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3929411873 wait complete. 2024-09-10T19:19:30.460 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6fe99640 1 Processor -- start 2024-09-10T19:19:30.460 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6fe99640 1 -- start start 2024-09-10T19:19:30.460 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6fe99640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5b680fce70 0x7f5b68112c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:30.460 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6fe99640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b681026d0 0x7f5b681131a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:30.460 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6fe99640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b68103070 0x7f5b6810ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:30.460 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6fe99640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6810d460 con 0x7f5b681026d0 2024-09-10T19:19:30.460 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6fe99640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6810d5d0 con 0x7f5b68103070 2024-09-10T19:19:30.461 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6fe99640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6810d790 con 0x7f5b680fce70 2024-09-10T19:19:30.461 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6dc35640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5b680fce70 0x7f5b68112c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:30.461 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6d434640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b681026d0 0x7f5b681131a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:30.461 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6dc35640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5b680fce70 0x7f5b68112c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:44106/0 (socket says 172.21.15.119:44106) 2024-09-10T19:19:30.461 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6dc35640 1 -- 172.21.15.119:0/3228563143 learned_addr learned my addr 172.21.15.119:0/3228563143 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:19:30.461 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6dc35640 1 -- 172.21.15.119:0/3228563143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b68103070 msgr2=0x7f5b6810ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:30.461 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6dc35640 1 --2- 172.21.15.119:0/3228563143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b68103070 0x7f5b6810ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:30.461 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6dc35640 1 -- 172.21.15.119:0/3228563143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b681026d0 msgr2=0x7f5b681131a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:30.461 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6dc35640 1 --2- 172.21.15.119:0/3228563143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b681026d0 0x7f5b681131a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:30.461 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6dc35640 1 -- 172.21.15.119:0/3228563143 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b64005040 con 0x7f5b680fce70 2024-09-10T19:19:30.461 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6d434640 1 --2- 172.21.15.119:0/3228563143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b681026d0 0x7f5b681131a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:19:30.462 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.456+0000 7f5b6dc35640 1 --2- 172.21.15.119:0/3228563143 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5b680fce70 0x7f5b68112c60 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7f5b64009550 tx=0x7f5b64009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:19:30.462 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.457+0000 7f5b5e7fc640 1 -- 172.21.15.119:0/3228563143 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6400e930 con 0x7f5b680fce70 2024-09-10T19:19:30.462 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.457+0000 7f5b5e7fc640 1 -- 172.21.15.119:0/3228563143 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f5b6400f040 con 0x7f5b680fce70 2024-09-10T19:19:30.462 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.457+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3228563143 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b58005040 con 0x7f5b680fce70 2024-09-10T19:19:30.462 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.457+0000 7f5b5e7fc640 1 -- 172.21.15.119:0/3228563143 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b64013c60 con 0x7f5b680fce70 2024-09-10T19:19:30.462 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.457+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3228563143 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b68119a60 con 0x7f5b680fce70 2024-09-10T19:19:30.462 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.458+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3228563143 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b680fde80 con 0x7f5b680fce70 2024-09-10T19:19:30.463 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.458+0000 7f5b5e7fc640 1 -- 172.21.15.119:0/3228563143 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f5b6400ea90 con 0x7f5b680fce70 2024-09-10T19:19:30.463 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.459+0000 7f5b5e7fc640 1 --2- 172.21.15.119:0/3228563143 >> v2:172.21.15.119:6800/3118961733 conn(0x7f5b5406ef80 0x7f5b54071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:19:30.463 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.459+0000 7f5b5e7fc640 1 -- 172.21.15.119:0/3228563143 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(758..758 src has 1..758) v4 ==== 8243+0+0 (secure 0 0 0) 0x7f5b6408eb90 con 0x7f5b680fce70 2024-09-10T19:19:30.463 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.459+0000 7f5b6d434640 1 --2- 172.21.15.119:0/3228563143 >> v2:172.21.15.119:6800/3118961733 conn(0x7f5b5406ef80 0x7f5b54071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:19:30.463 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.459+0000 7f5b6d434640 1 --2- 172.21.15.119:0/3228563143 >> v2:172.21.15.119:6800/3118961733 conn(0x7f5b5406ef80 0x7f5b54071430 secure :-1 s=READY pgs=3728 cs=0 l=1 rev1=1 crypto rx=0x7f5b58004890 tx=0x7f5b58035040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:19:30.464 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.463+0000 7f5b5e7fc640 1 -- 172.21.15.119:0/3228563143 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f5b6405b190 con 0x7f5b680fce70 2024-09-10T19:19:30.519 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:19:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:19:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:19:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:30 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:30 smithi194 ceph-mon[32464]: pgmap v1214: 188 pgs: 12 unknown, 176 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 1 op/s 2024-09-10T19:19:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:30 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]': finished 2024-09-10T19:19:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:30 smithi194 ceph-mon[32464]: osdmap e757: 8 total, 8 up, 8 in 2024-09-10T19:19:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:30 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1165874660' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]: dispatch 2024-09-10T19:19:30.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:30 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]: dispatch 2024-09-10T19:19:30.520 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:30 smithi194 ceph-mon[32464]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:19:30.620 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:30.618+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3228563143 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f5b6810bc30 con 0x7f5b680fce70 2024-09-10T19:19:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:30 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:30 smithi119 ceph-mon[29577]: pgmap v1214: 188 pgs: 12 unknown, 176 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 1 op/s 2024-09-10T19:19:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:30 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]': finished 2024-09-10T19:19:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:30 smithi119 ceph-mon[29577]: osdmap e757: 8 total, 8 up, 8 in 2024-09-10T19:19:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:30 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1165874660' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]: dispatch 2024-09-10T19:19:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:30 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]: dispatch 2024-09-10T19:19:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:30 smithi119 ceph-mon[29577]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:19:30.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:30 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:30.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:30 smithi119 ceph-mon[40674]: pgmap v1214: 188 pgs: 12 unknown, 176 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 1 op/s 2024-09-10T19:19:30.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:30 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]': finished 2024-09-10T19:19:30.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:30 smithi119 ceph-mon[40674]: osdmap e757: 8 total, 8 up, 8 in 2024-09-10T19:19:30.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:30 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1165874660' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]: dispatch 2024-09-10T19:19:30.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:30 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]: dispatch 2024-09-10T19:19:30.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:30 smithi119 ceph-mon[40674]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:19:31.371 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:31.369+0000 7f5b5e7fc640 1 -- 172.21.15.119:0/3228563143 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 0ceff89d-376b-4634-ba36-dca6344065fb v759) v1 ==== 223+0+0 (secure 0 0 0) 0x7f5b6405ab30 con 0x7f5b680fce70 2024-09-10T19:19:31.464 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:31.462+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3228563143 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f5b6810e2a0 con 0x7f5b680fce70 2024-09-10T19:19:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]': finished 2024-09-10T19:19:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:31 smithi194 ceph-mon[32464]: osdmap e758: 8 total, 8 up, 8 in 2024-09-10T19:19:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:31 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3228563143' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:19:31.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:31 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:19:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]': finished 2024-09-10T19:19:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:31 smithi119 ceph-mon[29577]: osdmap e758: 8 total, 8 up, 8 in 2024-09-10T19:19:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:31 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3228563143' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:19:31.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:31 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:19:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "app": "rados"}]': finished 2024-09-10T19:19:31.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:31 smithi119 ceph-mon[40674]: osdmap e758: 8 total, 8 up, 8 in 2024-09-10T19:19:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:31 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3228563143' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:19:31.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:31 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:19:32.374 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:32.372+0000 7f5b5e7fc640 1 -- 172.21.15.119:0/3228563143 <== mon.2 v2:172.21.15.119:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 0ceff89d-376b-4634-ba36-dca6344065fb v760) v1 ==== 223+0+0 (secure 0 0 0) 0x7f5b6405ad10 con 0x7f5b680fce70 2024-09-10T19:19:32.449 INFO:tasks.workunit.client.0.smithi119.stderr:set-quota max_objects = 10 for pool 0ceff89d-376b-4634-ba36-dca6344065fb 2024-09-10T19:19:32.450 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:32.375+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3228563143 >> v2:172.21.15.119:6800/3118961733 conn(0x7f5b5406ef80 msgr2=0x7f5b54071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:32.450 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:32.375+0000 7f5b6fe99640 1 --2- 172.21.15.119:0/3228563143 >> v2:172.21.15.119:6800/3118961733 conn(0x7f5b5406ef80 0x7f5b54071430 secure :-1 s=READY pgs=3728 cs=0 l=1 rev1=1 crypto rx=0x7f5b58004890 tx=0x7f5b58035040 comp rx=0 tx=0).stop 2024-09-10T19:19:32.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:32.375+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3228563143 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5b680fce70 msgr2=0x7f5b68112c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:19:32.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:32.375+0000 7f5b6fe99640 1 --2- 172.21.15.119:0/3228563143 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5b680fce70 0x7f5b68112c60 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7f5b64009550 tx=0x7f5b64009da0 comp rx=0 tx=0).stop 2024-09-10T19:19:32.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:32.375+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3228563143 shutdown_connections 2024-09-10T19:19:32.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:32.375+0000 7f5b6fe99640 1 --2- 172.21.15.119:0/3228563143 >> v2:172.21.15.119:6800/3118961733 conn(0x7f5b5406ef80 0x7f5b54071430 unknown :-1 s=CLOSED pgs=3728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:32.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:32.375+0000 7f5b6fe99640 1 --2- 172.21.15.119:0/3228563143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b68103070 0x7f5b6810ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:32.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:32.375+0000 7f5b6fe99640 1 --2- 172.21.15.119:0/3228563143 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5b681026d0 0x7f5b681131a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:32.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:32.375+0000 7f5b6fe99640 1 --2- 172.21.15.119:0/3228563143 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5b680fce70 0x7f5b68112c60 unknown :-1 s=CLOSED pgs=2957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:19:32.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:32.375+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3228563143 >> 172.21.15.119:0/3228563143 conn(0x7f5b680f82e0 msgr2=0x7f5b68100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:19:32.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:32.375+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3228563143 shutdown_connections 2024-09-10T19:19:32.451 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:19:32.375+0000 7f5b6fe99640 1 -- 172.21.15.119:0/3228563143 wait complete. 2024-09-10T19:19:32.451 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 30 2024-09-10T19:19:32.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-mon[40674]: pgmap v1217: 188 pgs: 12 unknown, 176 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 1 op/s 2024-09-10T19:19:32.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]': finished 2024-09-10T19:19:32.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-mon[40674]: osdmap e759: 8 total, 8 up, 8 in 2024-09-10T19:19:32.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3228563143' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:19:32.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:19:32.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:19:32.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:32.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-mon[29577]: pgmap v1217: 188 pgs: 12 unknown, 176 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 1 op/s 2024-09-10T19:19:32.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]': finished 2024-09-10T19:19:32.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-mon[29577]: osdmap e759: 8 total, 8 up, 8 in 2024-09-10T19:19:32.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3228563143' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:19:32.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:19:32.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:19:32.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:32 smithi194 ceph-mon[32464]: pgmap v1217: 188 pgs: 12 unknown, 176 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.5 KiB/s wr, 1 op/s 2024-09-10T19:19:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]': finished 2024-09-10T19:19:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:32 smithi194 ceph-mon[32464]: osdmap e759: 8 total, 8 up, 8 in 2024-09-10T19:19:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:32 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3228563143' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:19:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:32 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]: dispatch 2024-09-10T19:19:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:19:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:33.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:19:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:19:32] "GET /metrics HTTP/1.1" 200 228589 "" "Prometheus/2.43.0" 2024-09-10T19:19:33.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]': finished 2024-09-10T19:19:33.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:33 smithi194 ceph-mon[32464]: osdmap e760: 8 total, 8 up, 8 in 2024-09-10T19:19:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]': finished 2024-09-10T19:19:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:33 smithi119 ceph-mon[29577]: osdmap e760: 8 total, 8 up, 8 in 2024-09-10T19:19:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "field": "max_objects", "val": "10"}]': finished 2024-09-10T19:19:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:33 smithi119 ceph-mon[40674]: osdmap e760: 8 total, 8 up, 8 in 2024-09-10T19:19:34.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:34 smithi194 ceph-mon[32464]: pgmap v1220: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:34 smithi119 ceph-mon[29577]: pgmap v1220: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:34.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:34 smithi119 ceph-mon[40674]: pgmap v1220: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:36.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:36 smithi194 ceph-mon[32464]: pgmap v1221: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:36 smithi119 ceph-mon[29577]: pgmap v1221: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:36 smithi119 ceph-mon[40674]: pgmap v1221: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:37 smithi194 ceph-mon[32464]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:19:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:37 smithi119 ceph-mon[29577]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:19:37.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:37 smithi119 ceph-mon[40674]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:19:38.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:38 smithi194 ceph-mon[32464]: pgmap v1222: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-09-10T19:19:38.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:38 smithi119 ceph-mon[29577]: pgmap v1222: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-09-10T19:19:38.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:38 smithi119 ceph-mon[40674]: pgmap v1222: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-09-10T19:19:39.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:19:38 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:19:40.396 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:19:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:19:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:19:40.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:40 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:40.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:40 smithi194 ceph-mon[32464]: pgmap v1223: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:40 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:40 smithi119 ceph-mon[29577]: pgmap v1223: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:40.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:40 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:40.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:40 smithi119 ceph-mon[40674]: pgmap v1223: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:42.705 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:42 smithi119 ceph-mon[29577]: pgmap v1224: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:19:42.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:42 smithi119 ceph-mon[40674]: pgmap v1224: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:19:42.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:42 smithi194 ceph-mon[32464]: pgmap v1224: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:19:43.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:19:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:19:42] "GET /metrics HTTP/1.1" 200 228589 "" "Prometheus/2.43.0" 2024-09-10T19:19:44.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:44 smithi194 ceph-mon[32464]: pgmap v1225: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:19:44.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:44 smithi119 ceph-mon[29577]: pgmap v1225: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:19:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:44 smithi119 ceph-mon[40674]: pgmap v1225: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:19:46.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:46 smithi194 ceph-mon[32464]: pgmap v1226: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:19:46.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:46 smithi119 ceph-mon[29577]: pgmap v1226: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:19:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:46 smithi119 ceph-mon[40674]: pgmap v1226: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:19:47.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:47.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:47.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:48.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:48 smithi194 ceph-mon[32464]: pgmap v1227: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:48 smithi119 ceph-mon[29577]: pgmap v1227: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:48.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:48 smithi119 ceph-mon[40674]: pgmap v1227: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:49.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:19:49 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:19:50.416 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:19:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:19:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:19:50.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:50 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:50.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:50 smithi194 ceph-mon[32464]: pgmap v1228: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:50 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:50 smithi119 ceph-mon[29577]: pgmap v1228: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:50.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:50 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:50.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:50 smithi119 ceph-mon[40674]: pgmap v1228: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:52.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:52 smithi119 ceph-mon[29577]: pgmap v1229: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:19:52.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:52 smithi119 ceph-mon[40674]: pgmap v1229: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:19:52.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:52 smithi194 ceph-mon[32464]: pgmap v1229: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:19:53.068 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:19:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:19:52] "GET /metrics HTTP/1.1" 200 228600 "" "Prometheus/2.43.0" 2024-09-10T19:19:53.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:53 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:19:53.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:53 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:19:53.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:53 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:19:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:54 smithi194 ceph-mon[32464]: pgmap v1230: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:54 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:19:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:54 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:19:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:54 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:19:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:54 smithi119 ceph-mon[29577]: pgmap v1230: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:54 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:19:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:54 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:19:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:54 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:19:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:54 smithi119 ceph-mon[40674]: pgmap v1230: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:19:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:54 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:19:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:54 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:19:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:54 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:19:56.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:56 smithi194 ceph-mon[32464]: pgmap v1231: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:56.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:56 smithi119 ceph-mon[29577]: pgmap v1231: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:56 smithi119 ceph-mon[40674]: pgmap v1231: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:58.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:19:58 smithi194 ceph-mon[32464]: pgmap v1232: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:19:58 smithi119 ceph-mon[29577]: pgmap v1232: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:58.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:19:58 smithi119 ceph-mon[40674]: pgmap v1232: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:59.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:19:59 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:20:00.437 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:20:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:20:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:20:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:00 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:00 smithi194 ceph-mon[32464]: pgmap v1233: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:00 smithi194 ceph-mon[32464]: overall HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-10T19:20:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:00 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:00 smithi119 ceph-mon[29577]: pgmap v1233: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:00 smithi119 ceph-mon[29577]: overall HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-10T19:20:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:00 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:00 smithi119 ceph-mon[40674]: pgmap v1233: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:00 smithi119 ceph-mon[40674]: overall HEALTH_WARN 1 pool(s) do not have an application enabled 2024-09-10T19:20:02.389 INFO:tasks.workunit.client.0.smithi119.stderr:++ seq 1 10 2024-09-10T19:20:02.390 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:20:02.390 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 0ceff89d-376b-4634-ba36-dca6344065fb put obj1 /etc/passwd 2024-09-10T19:20:02.428 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:20:02.428 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 0ceff89d-376b-4634-ba36-dca6344065fb put obj2 /etc/passwd 2024-09-10T19:20:02.461 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:20:02.462 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 0ceff89d-376b-4634-ba36-dca6344065fb put obj3 /etc/passwd 2024-09-10T19:20:02.494 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:20:02.494 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 0ceff89d-376b-4634-ba36-dca6344065fb put obj4 /etc/passwd 2024-09-10T19:20:02.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:20:02.525 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 0ceff89d-376b-4634-ba36-dca6344065fb put obj5 /etc/passwd 2024-09-10T19:20:02.556 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:20:02.556 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 0ceff89d-376b-4634-ba36-dca6344065fb put obj6 /etc/passwd 2024-09-10T19:20:02.586 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:20:02.586 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 0ceff89d-376b-4634-ba36-dca6344065fb put obj7 /etc/passwd 2024-09-10T19:20:02.616 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:20:02.617 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 0ceff89d-376b-4634-ba36-dca6344065fb put obj8 /etc/passwd 2024-09-10T19:20:02.647 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:20:02.647 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 0ceff89d-376b-4634-ba36-dca6344065fb put obj9 /etc/passwd 2024-09-10T19:20:02.677 INFO:tasks.workunit.client.0.smithi119.stderr:+ for f in `seq 1 10` 2024-09-10T19:20:02.678 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 0ceff89d-376b-4634-ba36-dca6344065fb put obj10 /etc/passwd 2024-09-10T19:20:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:02 smithi119 ceph-mon[29577]: pgmap v1234: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:02.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:02 smithi119 ceph-mon[40674]: pgmap v1234: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:02.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:02.708 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 30 2024-09-10T19:20:02.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:02 smithi194 ceph-mon[32464]: pgmap v1234: 188 pgs: 188 active+clean; 757 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:02.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:03.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:20:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:20:02] "GET /metrics HTTP/1.1" 200 228579 "" "Prometheus/2.43.0" 2024-09-10T19:20:04.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:04 smithi194 ceph-mon[32464]: pgmap v1235: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 2 op/s 2024-09-10T19:20:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:04 smithi119 ceph-mon[29577]: pgmap v1235: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 2 op/s 2024-09-10T19:20:04.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:04 smithi119 ceph-mon[40674]: pgmap v1235: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 11 KiB/s wr, 2 op/s 2024-09-10T19:20:06.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:06 smithi194 ceph-mon[32464]: pgmap v1236: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 11 KiB/s wr, 1 op/s 2024-09-10T19:20:06.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:06 smithi119 ceph-mon[29577]: pgmap v1236: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 11 KiB/s wr, 1 op/s 2024-09-10T19:20:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:06 smithi119 ceph-mon[40674]: pgmap v1236: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 11 KiB/s wr, 1 op/s 2024-09-10T19:20:07.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:07 smithi194 ceph-mon[32464]: pool '0ceff89d-376b-4634-ba36-dca6344065fb' is full (reached quota's max_objects: 10) 2024-09-10T19:20:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:07 smithi194 ceph-mon[32464]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T19:20:07.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:07 smithi194 ceph-mon[32464]: osdmap e761: 8 total, 8 up, 8 in 2024-09-10T19:20:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:07 smithi119 ceph-mon[29577]: pool '0ceff89d-376b-4634-ba36-dca6344065fb' is full (reached quota's max_objects: 10) 2024-09-10T19:20:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:07 smithi119 ceph-mon[29577]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T19:20:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:07 smithi119 ceph-mon[29577]: osdmap e761: 8 total, 8 up, 8 in 2024-09-10T19:20:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:07 smithi119 ceph-mon[40674]: pool '0ceff89d-376b-4634-ba36-dca6344065fb' is full (reached quota's max_objects: 10) 2024-09-10T19:20:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:07 smithi119 ceph-mon[40674]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-09-10T19:20:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:07 smithi119 ceph-mon[40674]: osdmap e761: 8 total, 8 up, 8 in 2024-09-10T19:20:08.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:08 smithi194 ceph-mon[32464]: pgmap v1238: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:20:08.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:08 smithi119 ceph-mon[29577]: pgmap v1238: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:20:08.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:08 smithi119 ceph-mon[40674]: pgmap v1238: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:20:09.519 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:20:09 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:20:10.458 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:20:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:20:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:20:10.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:10 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:10.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:10 smithi194 ceph-mon[32464]: pgmap v1239: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:20:10.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:10 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:10.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:10 smithi119 ceph-mon[29577]: pgmap v1239: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:20:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:10 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:10.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:10 smithi119 ceph-mon[40674]: pgmap v1239: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:20:12.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:12 smithi194 ceph-mon[32464]: pgmap v1240: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:20:12.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:12 smithi119 ceph-mon[29577]: pgmap v1240: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:20:12.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:12 smithi119 ceph-mon[40674]: pgmap v1240: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 KiB/s wr, 1 op/s 2024-09-10T19:20:12.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:20:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:20:12] "GET /metrics HTTP/1.1" 200 228579 "" "Prometheus/2.43.0" 2024-09-10T19:20:14.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:14 smithi194 ceph-mon[32464]: pgmap v1241: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:20:14.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:14 smithi119 ceph-mon[29577]: pgmap v1241: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:20:14.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:14 smithi119 ceph-mon[40674]: pgmap v1241: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:20:16.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:16 smithi194 ceph-mon[32464]: pgmap v1242: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:20:16.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:16 smithi119 ceph-mon[29577]: pgmap v1242: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:20:16.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:16 smithi119 ceph-mon[40674]: pgmap v1242: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:20:18.518 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:18 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:20:18.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:18 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:18.519 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:18 smithi194 ceph-mon[32464]: pgmap v1243: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-09-10T19:20:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:18 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:20:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:18 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:18.567 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:18 smithi119 ceph-mon[29577]: pgmap v1243: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-09-10T19:20:18.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:18 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:20:18.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:18 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:18.567 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:18 smithi119 ceph-mon[40674]: pgmap v1243: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-09-10T19:20:19.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:20:19 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:20:20.416 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:20:20 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:20:20] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:20:20.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:20 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:20.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:20 smithi194 ceph-mon[32464]: pgmap v1244: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:20 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:20.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:20 smithi119 ceph-mon[29577]: pgmap v1244: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:20 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:20.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:20 smithi119 ceph-mon[40674]: pgmap v1244: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:22.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:22 smithi119 ceph-mon[29577]: pgmap v1245: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:22.705 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:22 smithi119 ceph-mon[40674]: pgmap v1245: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:22.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:22 smithi194 ceph-mon[32464]: pgmap v1245: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:23.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:20:22 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:20:22] "GET /metrics HTTP/1.1" 200 228674 "" "Prometheus/2.43.0" 2024-09-10T19:20:24.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:24 smithi194 ceph-mon[32464]: pgmap v1246: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:24.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:24 smithi119 ceph-mon[29577]: pgmap v1246: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:24.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:24 smithi119 ceph-mon[40674]: pgmap v1246: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:26.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:26 smithi194 ceph-mon[32464]: pgmap v1247: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:26.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:26 smithi119 ceph-mon[29577]: pgmap v1247: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:26.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:26 smithi119 ceph-mon[40674]: pgmap v1247: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:28.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:28 smithi194 ceph-mon[32464]: pgmap v1248: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:28.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:28 smithi119 ceph-mon[29577]: pgmap v1248: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:28.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:28 smithi119 ceph-mon[40674]: pgmap v1248: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:29.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:20:29 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:20:30.420 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:20:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:20:30] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:20:30.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:30 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:30.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:30 smithi194 ceph-mon[32464]: pgmap v1249: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:30.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:30 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:30.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:30 smithi119 ceph-mon[29577]: pgmap v1249: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:30 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:30.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:30 smithi119 ceph-mon[40674]: pgmap v1249: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:32.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:32 smithi119 ceph-mon[29577]: pgmap v1250: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:20:32.704 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:32 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:32.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:32 smithi119 ceph-mon[40674]: pgmap v1250: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:20:32.704 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:32 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:32.709 INFO:tasks.workunit.client.0.smithi119.stderr:+ rados -p 791ac3f1-f414-442d-8950-49875e1f19bf put threemore /etc/passwd 2024-09-10T19:20:32.745 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd pool set-quota 791ac3f1-f414-442d-8950-49875e1f19bf max_bytes 0 2024-09-10T19:20:32.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:32 smithi194 ceph-mon[32464]: pgmap v1250: 188 pgs: 188 active+clean; 882 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:20:32.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:32 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:32.830 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.826+0000 7f6de9c34640 1 -- 172.21.15.119:0/1867221484 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6de40fd380 msgr2=0x7f6de41094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:20:32.830 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.826+0000 7f6de9c34640 1 --2- 172.21.15.119:0/1867221484 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6de40fd380 0x7f6de41094f0 secure :-1 s=READY pgs=2974 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009d20 tx=0x7f6ddc004c50 comp rx=0 tx=0).stop 2024-09-10T19:20:32.830 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.826+0000 7f6de9c34640 1 -- 172.21.15.119:0/1867221484 shutdown_connections 2024-09-10T19:20:32.830 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.826+0000 7f6de9c34640 1 --2- 172.21.15.119:0/1867221484 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6de40fd380 0x7f6de41094f0 unknown :-1 s=CLOSED pgs=2974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:32.830 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.826+0000 7f6de9c34640 1 --2- 172.21.15.119:0/1867221484 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de40fc9f0 0x7f6de40fce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:32.830 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.826+0000 7f6de9c34640 1 --2- 172.21.15.119:0/1867221484 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6de4102760 0x7f6de4102b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:32.831 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.826+0000 7f6de9c34640 1 -- 172.21.15.119:0/1867221484 >> 172.21.15.119:0/1867221484 conn(0x7f6de4074870 msgr2=0x7f6de4074c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:20:32.831 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.826+0000 7f6de9c34640 1 -- 172.21.15.119:0/1867221484 shutdown_connections 2024-09-10T19:20:32.831 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.826+0000 7f6de9c34640 1 -- 172.21.15.119:0/1867221484 wait complete. 2024-09-10T19:20:32.831 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de9c34640 1 Processor -- start 2024-09-10T19:20:32.831 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de9c34640 1 -- start start 2024-09-10T19:20:32.831 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de9c34640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6de40fc9f0 0x7f6de4112930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:20:32.831 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de9c34640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6de40fd380 0x7f6de4112e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:20:32.831 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de9c34640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de4102760 0x7f6de410caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:20:32.831 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de9c34640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de410d140 con 0x7f6de4102760 2024-09-10T19:20:32.831 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de9c34640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de410d2b0 con 0x7f6de40fd380 2024-09-10T19:20:32.831 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de9c34640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de410d470 con 0x7f6de40fc9f0 2024-09-10T19:20:32.831 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de37fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6de40fc9f0 0x7f6de4112930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:20:32.832 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de37fe640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6de40fc9f0 0x7f6de4112930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:45734/0 (socket says 172.21.15.119:45734) 2024-09-10T19:20:32.832 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de37fe640 1 -- 172.21.15.119:0/963862756 learned_addr learned my addr 172.21.15.119:0/963862756 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:20:32.832 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de3fff640 1 --2- 172.21.15.119:0/963862756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de4102760 0x7f6de410caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:20:32.832 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6ddbfff640 1 --2- 172.21.15.119:0/963862756 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6de40fd380 0x7f6de4112e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:20:32.832 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de37fe640 1 -- 172.21.15.119:0/963862756 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6de40fd380 msgr2=0x7f6de4112e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:20:32.832 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de37fe640 1 --2- 172.21.15.119:0/963862756 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6de40fd380 0x7f6de4112e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:32.832 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de37fe640 1 -- 172.21.15.119:0/963862756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de4102760 msgr2=0x7f6de410caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:20:32.832 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de37fe640 1 --2- 172.21.15.119:0/963862756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de4102760 0x7f6de410caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:32.832 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de37fe640 1 -- 172.21.15.119:0/963862756 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc005040 con 0x7f6de40fc9f0 2024-09-10T19:20:32.832 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de3fff640 1 --2- 172.21.15.119:0/963862756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de4102760 0x7f6de410caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:20:32.832 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6ddbfff640 1 --2- 172.21.15.119:0/963862756 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6de40fd380 0x7f6de4112e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T19:20:32.832 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.827+0000 7f6de37fe640 1 --2- 172.21.15.119:0/963862756 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6de40fc9f0 0x7f6de4112930 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4002ad0 tx=0x7f6dd4004600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:20:32.833 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.828+0000 7f6de0ff9640 1 -- 172.21.15.119:0/963862756 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4007c30 con 0x7f6de40fc9f0 2024-09-10T19:20:32.833 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.828+0000 7f6de0ff9640 1 -- 172.21.15.119:0/963862756 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f6dd400f040 con 0x7f6de40fc9f0 2024-09-10T19:20:32.833 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.828+0000 7f6de9c34640 1 -- 172.21.15.119:0/963862756 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ddc0098b0 con 0x7f6de40fc9f0 2024-09-10T19:20:32.833 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.828+0000 7f6de0ff9640 1 -- 172.21.15.119:0/963862756 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4009340 con 0x7f6de40fc9f0 2024-09-10T19:20:32.833 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.828+0000 7f6de9c34640 1 -- 172.21.15.119:0/963862756 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de4119750 con 0x7f6de40fc9f0 2024-09-10T19:20:32.833 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.829+0000 7f6de9c34640 1 -- 172.21.15.119:0/963862756 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de40fde80 con 0x7f6de40fc9f0 2024-09-10T19:20:32.834 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.829+0000 7f6de0ff9640 1 -- 172.21.15.119:0/963862756 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f6dd40096f0 con 0x7f6de40fc9f0 2024-09-10T19:20:32.834 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.830+0000 7f6de0ff9640 1 --2- 172.21.15.119:0/963862756 >> v2:172.21.15.119:6800/3118961733 conn(0x7f6dd006ef80 0x7f6dd0071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:20:32.834 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.830+0000 7f6de0ff9640 1 -- 172.21.15.119:0/963862756 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(761..761 src has 1..761) v4 ==== 8243+0+0 (secure 0 0 0) 0x7f6dd408eaf0 con 0x7f6de40fc9f0 2024-09-10T19:20:32.834 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.830+0000 7f6ddbfff640 1 --2- 172.21.15.119:0/963862756 >> v2:172.21.15.119:6800/3118961733 conn(0x7f6dd006ef80 0x7f6dd0071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:20:32.834 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.830+0000 7f6de0ff9640 1 -- 172.21.15.119:0/963862756 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=762}) v3 -- 0x7f6dd007ab60 con 0x7f6de40fc9f0 2024-09-10T19:20:32.834 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.830+0000 7f6ddbfff640 1 --2- 172.21.15.119:0/963862756 >> v2:172.21.15.119:6800/3118961733 conn(0x7f6dd006ef80 0x7f6dd0071430 secure :-1 s=READY pgs=3740 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc007fd0 tx=0x7f6dcc009da0 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:20:32.835 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.834+0000 7f6de0ff9640 1 -- 172.21.15.119:0/963862756 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f6dd405b1b0 con 0x7f6de40fc9f0 2024-09-10T19:20:32.991 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:32.990+0000 7f6de9c34640 1 -- 172.21.15.119:0/963862756 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f6de410dfa0 con 0x7f6de40fc9f0 2024-09-10T19:20:33.066 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:20:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:20:32] "GET /metrics HTTP/1.1" 200 228647 "" "Prometheus/2.43.0" 2024-09-10T19:20:33.435 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:33.433+0000 7f6de0ff9640 1 -- 172.21.15.119:0/963862756 <== mon.2 v2:172.21.15.119:3301/0 7 ==== osd_map(762..762 src has 1..762) v4 ==== 620+0+0 (secure 0 0 0) 0x7f6dd4007d90 con 0x7f6de40fc9f0 2024-09-10T19:20:33.435 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:33.433+0000 7f6de0ff9640 1 -- 172.21.15.119:0/963862756 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=763}) v3 -- 0x7f6dd007b770 con 0x7f6de40fc9f0 2024-09-10T19:20:33.439 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:33.437+0000 7f6de0ff9640 1 -- 172.21.15.119:0/963862756 <== mon.2 v2:172.21.15.119:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 791ac3f1-f414-442d-8950-49875e1f19bf v762) v1 ==== 217+0+0 (secure 0 0 0) 0x7f6dd405ab50 con 0x7f6de40fc9f0 2024-09-10T19:20:33.533 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:33.531+0000 7f6de9c34640 1 -- 172.21.15.119:0/963862756 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f6de410bb80 con 0x7f6de40fc9f0 2024-09-10T19:20:33.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:33 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/963862756' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:20:33.794 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:33 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:20:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:33 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/963862756' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:20:33.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:33 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:20:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:33 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/963862756' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:20:33.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:33 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:20:34.436 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.435+0000 7f6de0ff9640 1 -- 172.21.15.119:0/963862756 <== mon.2 v2:172.21.15.119:3301/0 9 ==== osd_map(763..763 src has 1..763) v4 ==== 620+0+0 (secure 0 0 0) 0x7f6dd405aef0 con 0x7f6de40fc9f0 2024-09-10T19:20:34.437 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.435+0000 7f6de0ff9640 1 -- 172.21.15.119:0/963862756 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=764}) v3 -- 0x7f6dd007bc80 con 0x7f6de40fc9f0 2024-09-10T19:20:34.440 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.439+0000 7f6de0ff9640 1 -- 172.21.15.119:0/963862756 <== mon.2 v2:172.21.15.119:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 791ac3f1-f414-442d-8950-49875e1f19bf v763) v1 ==== 217+0+0 (secure 0 0 0) 0x7f6dd4051020 con 0x7f6de40fc9f0 2024-09-10T19:20:34.441 INFO:tasks.workunit.client.0.smithi119.stderr:set-quota max_bytes = 0 for pool 791ac3f1-f414-442d-8950-49875e1f19bf 2024-09-10T19:20:34.445 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.443+0000 7f6de9c34640 1 -- 172.21.15.119:0/963862756 >> v2:172.21.15.119:6800/3118961733 conn(0x7f6dd006ef80 msgr2=0x7f6dd0071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:20:34.445 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.443+0000 7f6de9c34640 1 --2- 172.21.15.119:0/963862756 >> v2:172.21.15.119:6800/3118961733 conn(0x7f6dd006ef80 0x7f6dd0071430 secure :-1 s=READY pgs=3740 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc007fd0 tx=0x7f6dcc009da0 comp rx=0 tx=0).stop 2024-09-10T19:20:34.445 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.443+0000 7f6de9c34640 1 -- 172.21.15.119:0/963862756 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6de40fc9f0 msgr2=0x7f6de4112930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:20:34.445 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.443+0000 7f6de9c34640 1 --2- 172.21.15.119:0/963862756 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6de40fc9f0 0x7f6de4112930 secure :-1 s=READY pgs=2975 cs=0 l=1 rev1=1 crypto rx=0x7f6dd4002ad0 tx=0x7f6dd4004600 comp rx=0 tx=0).stop 2024-09-10T19:20:34.445 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.443+0000 7f6de9c34640 1 -- 172.21.15.119:0/963862756 shutdown_connections 2024-09-10T19:20:34.445 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.443+0000 7f6de9c34640 1 --2- 172.21.15.119:0/963862756 >> v2:172.21.15.119:6800/3118961733 conn(0x7f6dd006ef80 0x7f6dd0071430 unknown :-1 s=CLOSED pgs=3740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:34.445 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.443+0000 7f6de9c34640 1 --2- 172.21.15.119:0/963862756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6de4102760 0x7f6de410caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:34.445 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.443+0000 7f6de9c34640 1 --2- 172.21.15.119:0/963862756 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6de40fd380 0x7f6de4112e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:34.445 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.443+0000 7f6de9c34640 1 --2- 172.21.15.119:0/963862756 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f6de40fc9f0 0x7f6de4112930 unknown :-1 s=CLOSED pgs=2975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:34.446 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.443+0000 7f6de9c34640 1 -- 172.21.15.119:0/963862756 >> 172.21.15.119:0/963862756 conn(0x7f6de4074870 msgr2=0x7f6de40fb270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:20:34.446 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.443+0000 7f6de9c34640 1 -- 172.21.15.119:0/963862756 shutdown_connections 2024-09-10T19:20:34.446 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.443+0000 7f6de9c34640 1 -- 172.21.15.119:0/963862756 wait complete. 2024-09-10T19:20:34.455 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd pool set-quota 791ac3f1-f414-442d-8950-49875e1f19bf max_objects 0 2024-09-10T19:20:34.537 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.533+0000 7fe753f09640 1 -- 172.21.15.119:0/1418650372 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe74c0fd0d0 msgr2=0x7fe74c1094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:20:34.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.533+0000 7fe753f09640 1 --2- 172.21.15.119:0/1418650372 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe74c0fd0d0 0x7fe74c1094f0 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7fe744009d00 tx=0x7fe744004d10 comp rx=0 tx=0).stop 2024-09-10T19:20:34.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.533+0000 7fe753f09640 1 -- 172.21.15.119:0/1418650372 shutdown_connections 2024-09-10T19:20:34.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.533+0000 7fe753f09640 1 --2- 172.21.15.119:0/1418650372 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe74c0fd0d0 0x7fe74c1094f0 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:34.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.533+0000 7fe753f09640 1 --2- 172.21.15.119:0/1418650372 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe74c0fc720 0x7fe74c0fcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:34.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.533+0000 7fe753f09640 1 --2- 172.21.15.119:0/1418650372 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe74c102740 0x7fe74c102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:34.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.533+0000 7fe753f09640 1 -- 172.21.15.119:0/1418650372 >> 172.21.15.119:0/1418650372 conn(0x7fe74c0f82c0 msgr2=0x7fe74c0fa6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:20:34.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.533+0000 7fe753f09640 1 -- 172.21.15.119:0/1418650372 shutdown_connections 2024-09-10T19:20:34.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.533+0000 7fe753f09640 1 -- 172.21.15.119:0/1418650372 wait complete. 2024-09-10T19:20:34.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe753f09640 1 Processor -- start 2024-09-10T19:20:34.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe753f09640 1 -- start start 2024-09-10T19:20:34.538 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe753f09640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe74c0fc720 0x7fe74c112960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:20:34.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe753f09640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe74c0fd0d0 0x7fe74c112ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:20:34.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe753f09640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe74c102740 0x7fe74c10cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:20:34.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe753f09640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c10d160 con 0x7fe74c102740 2024-09-10T19:20:34.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe753f09640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c10d2d0 con 0x7fe74c0fd0d0 2024-09-10T19:20:34.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe753f09640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c10d490 con 0x7fe74c0fc720 2024-09-10T19:20:34.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe751ca5640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe74c0fc720 0x7fe74c112960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:20:34.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe751ca5640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe74c0fc720 0x7fe74c112960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:58094/0 (socket says 172.21.15.119:58094) 2024-09-10T19:20:34.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe7524a6640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe74c102740 0x7fe74c10cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:20:34.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe751ca5640 1 -- 172.21.15.119:0/1945519470 learned_addr learned my addr 172.21.15.119:0/1945519470 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:20:34.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe7514a4640 1 --2- 172.21.15.119:0/1945519470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe74c0fd0d0 0x7fe74c112ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:20:34.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe751ca5640 1 -- 172.21.15.119:0/1945519470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe74c0fd0d0 msgr2=0x7fe74c112ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:20:34.539 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe751ca5640 1 --2- 172.21.15.119:0/1945519470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe74c0fd0d0 0x7fe74c112ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:34.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe751ca5640 1 -- 172.21.15.119:0/1945519470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe74c102740 msgr2=0x7fe74c10cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:20:34.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe751ca5640 1 --2- 172.21.15.119:0/1945519470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe74c102740 0x7fe74c10cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:34.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe751ca5640 1 -- 172.21.15.119:0/1945519470 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73c005040 con 0x7fe74c0fc720 2024-09-10T19:20:34.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe7514a4640 1 --2- 172.21.15.119:0/1945519470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe74c0fd0d0 0x7fe74c112ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T19:20:34.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe7524a6640 1 --2- 172.21.15.119:0/1945519470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe74c102740 0x7fe74c10cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:20:34.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe751ca5640 1 --2- 172.21.15.119:0/1945519470 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe74c0fc720 0x7fe74c112960 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c8b0 tx=0x7fe748004430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:20:34.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.534+0000 7fe7427fc640 1 -- 172.21.15.119:0/1945519470 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74800cea0 con 0x7fe74c0fc720 2024-09-10T19:20:34.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.535+0000 7fe7427fc640 1 -- 172.21.15.119:0/1945519470 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7fe74800acc0 con 0x7fe74c0fc720 2024-09-10T19:20:34.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.535+0000 7fe7427fc640 1 -- 172.21.15.119:0/1945519470 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748012bf0 con 0x7fe74c0fc720 2024-09-10T19:20:34.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.535+0000 7fe753f09640 1 -- 172.21.15.119:0/1945519470 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7440098b0 con 0x7fe74c0fc720 2024-09-10T19:20:34.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.535+0000 7fe753f09640 1 -- 172.21.15.119:0/1945519470 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74c119780 con 0x7fe74c0fc720 2024-09-10T19:20:34.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.536+0000 7fe753f09640 1 -- 172.21.15.119:0/1945519470 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe74c0fde60 con 0x7fe74c0fc720 2024-09-10T19:20:34.540 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.536+0000 7fe7427fc640 1 -- 172.21.15.119:0/1945519470 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7fe74800a850 con 0x7fe74c0fc720 2024-09-10T19:20:34.541 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.537+0000 7fe7427fc640 1 --2- 172.21.15.119:0/1945519470 >> v2:172.21.15.119:6800/3118961733 conn(0x7fe73806ef80 0x7fe738071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:20:34.541 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.537+0000 7fe7514a4640 1 --2- 172.21.15.119:0/1945519470 >> v2:172.21.15.119:6800/3118961733 conn(0x7fe73806ef80 0x7fe738071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:20:34.542 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.537+0000 7fe7427fc640 1 -- 172.21.15.119:0/1945519470 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(763..763 src has 1..763) v4 ==== 8243+0+0 (secure 0 0 0) 0x7fe74808ecb0 con 0x7fe74c0fc720 2024-09-10T19:20:34.542 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.537+0000 7fe7427fc640 1 -- 172.21.15.119:0/1945519470 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=764}) v3 -- 0x7fe73807ab60 con 0x7fe74c0fc720 2024-09-10T19:20:34.542 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.537+0000 7fe7514a4640 1 --2- 172.21.15.119:0/1945519470 >> v2:172.21.15.119:6800/3118961733 conn(0x7fe73806ef80 0x7fe738071430 secure :-1 s=READY pgs=3741 cs=0 l=1 rev1=1 crypto rx=0x7fe74c10ddf0 tx=0x7fe73c009da0 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:20:34.542 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.541+0000 7fe7427fc640 1 -- 172.21.15.119:0/1945519470 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7fe74805b330 con 0x7fe74c0fc720 2024-09-10T19:20:34.698 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:34.696+0000 7fe753f09640 1 -- 172.21.15.119:0/1945519470 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fe74c10dfd0 con 0x7fe74c0fc720 2024-09-10T19:20:34.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:34 smithi194 ceph-mon[32464]: pgmap v1251: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.1 KiB/s wr, 1 op/s 2024-09-10T19:20:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]': finished 2024-09-10T19:20:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:34 smithi194 ceph-mon[32464]: osdmap e762: 8 total, 8 up, 8 in 2024-09-10T19:20:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:34 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/963862756' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:20:34.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:34 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:20:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:34 smithi119 ceph-mon[29577]: pgmap v1251: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.1 KiB/s wr, 1 op/s 2024-09-10T19:20:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]': finished 2024-09-10T19:20:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:34 smithi119 ceph-mon[29577]: osdmap e762: 8 total, 8 up, 8 in 2024-09-10T19:20:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:34 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/963862756' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:20:34.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:34 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:20:34.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:34 smithi119 ceph-mon[40674]: pgmap v1251: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.1 KiB/s wr, 1 op/s 2024-09-10T19:20:34.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]': finished 2024-09-10T19:20:34.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:34 smithi119 ceph-mon[40674]: osdmap e762: 8 total, 8 up, 8 in 2024-09-10T19:20:34.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:34 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/963862756' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:20:34.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:34 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]: dispatch 2024-09-10T19:20:35.453 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:35.451+0000 7fe7427fc640 1 -- 172.21.15.119:0/1945519470 <== mon.2 v2:172.21.15.119:3301/0 7 ==== osd_map(764..764 src has 1..764) v4 ==== 620+0+0 (secure 0 0 0) 0x7fe74800ab00 con 0x7fe74c0fc720 2024-09-10T19:20:35.453 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:35.451+0000 7fe7427fc640 1 -- 172.21.15.119:0/1945519470 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=765}) v3 -- 0x7fe73807b710 con 0x7fe74c0fc720 2024-09-10T19:20:35.457 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:35.456+0000 7fe7427fc640 1 -- 172.21.15.119:0/1945519470 <== mon.2 v2:172.21.15.119:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 791ac3f1-f414-442d-8950-49875e1f19bf v764) v1 ==== 221+0+0 (secure 0 0 0) 0x7fe74805acd0 con 0x7fe74c0fc720 2024-09-10T19:20:35.555 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:35.554+0000 7fe753f09640 1 -- 172.21.15.119:0/1945519470 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fe74c10dfd0 con 0x7fe74c0fc720 2024-09-10T19:20:35.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:35 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]': finished 2024-09-10T19:20:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:35 smithi194 ceph-mon[32464]: osdmap e763: 8 total, 8 up, 8 in 2024-09-10T19:20:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:35 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1945519470' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:20:35.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:35 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:20:35.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:35 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]': finished 2024-09-10T19:20:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:35 smithi119 ceph-mon[29577]: osdmap e763: 8 total, 8 up, 8 in 2024-09-10T19:20:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:35 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1945519470' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:20:35.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:35 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:20:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:35 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_bytes", "val": "0"}]': finished 2024-09-10T19:20:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:35 smithi119 ceph-mon[40674]: osdmap e763: 8 total, 8 up, 8 in 2024-09-10T19:20:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:35 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1945519470' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:20:35.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:35 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:20:36.455 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.453+0000 7fe7427fc640 1 -- 172.21.15.119:0/1945519470 <== mon.2 v2:172.21.15.119:3301/0 9 ==== osd_map(765..765 src has 1..765) v4 ==== 620+0+0 (secure 0 0 0) 0x7fe748012400 con 0x7fe74c0fc720 2024-09-10T19:20:36.455 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.453+0000 7fe7427fc640 1 -- 172.21.15.119:0/1945519470 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=766}) v3 -- 0x7fe73807bc20 con 0x7fe74c0fc720 2024-09-10T19:20:36.459 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.458+0000 7fe7427fc640 1 -- 172.21.15.119:0/1945519470 <== mon.2 v2:172.21.15.119:3301/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 791ac3f1-f414-442d-8950-49875e1f19bf v765) v1 ==== 221+0+0 (secure 0 0 0) 0x7fe74805aeb0 con 0x7fe74c0fc720 2024-09-10T19:20:36.459 INFO:tasks.workunit.client.0.smithi119.stderr:set-quota max_objects = 0 for pool 791ac3f1-f414-442d-8950-49875e1f19bf 2024-09-10T19:20:36.463 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.461+0000 7fe753f09640 1 -- 172.21.15.119:0/1945519470 >> v2:172.21.15.119:6800/3118961733 conn(0x7fe73806ef80 msgr2=0x7fe738071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:20:36.463 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.461+0000 7fe753f09640 1 --2- 172.21.15.119:0/1945519470 >> v2:172.21.15.119:6800/3118961733 conn(0x7fe73806ef80 0x7fe738071430 secure :-1 s=READY pgs=3741 cs=0 l=1 rev1=1 crypto rx=0x7fe74c10ddf0 tx=0x7fe73c009da0 comp rx=0 tx=0).stop 2024-09-10T19:20:36.464 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.461+0000 7fe753f09640 1 -- 172.21.15.119:0/1945519470 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe74c0fc720 msgr2=0x7fe74c112960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:20:36.464 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.461+0000 7fe753f09640 1 --2- 172.21.15.119:0/1945519470 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe74c0fc720 0x7fe74c112960 secure :-1 s=READY pgs=2977 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c8b0 tx=0x7fe748004430 comp rx=0 tx=0).stop 2024-09-10T19:20:36.464 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.461+0000 7fe753f09640 1 -- 172.21.15.119:0/1945519470 shutdown_connections 2024-09-10T19:20:36.464 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.461+0000 7fe753f09640 1 --2- 172.21.15.119:0/1945519470 >> v2:172.21.15.119:6800/3118961733 conn(0x7fe73806ef80 0x7fe738071430 unknown :-1 s=CLOSED pgs=3741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:36.464 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.461+0000 7fe753f09640 1 --2- 172.21.15.119:0/1945519470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe74c102740 0x7fe74c10cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:36.464 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.461+0000 7fe753f09640 1 --2- 172.21.15.119:0/1945519470 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe74c0fd0d0 0x7fe74c112ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:36.464 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.461+0000 7fe753f09640 1 --2- 172.21.15.119:0/1945519470 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fe74c0fc720 0x7fe74c112960 unknown :-1 s=CLOSED pgs=2977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:20:36.464 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.461+0000 7fe753f09640 1 -- 172.21.15.119:0/1945519470 >> 172.21.15.119:0/1945519470 conn(0x7fe74c0f82c0 msgr2=0x7fe74c0f9680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:20:36.464 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.461+0000 7fe753f09640 1 -- 172.21.15.119:0/1945519470 shutdown_connections 2024-09-10T19:20:36.464 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:20:36.461+0000 7fe753f09640 1 -- 172.21.15.119:0/1945519470 wait complete. 2024-09-10T19:20:36.475 INFO:tasks.workunit.client.0.smithi119.stderr:+ sleep 30 2024-09-10T19:20:36.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:36 smithi194 ceph-mon[32464]: pgmap v1254: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-09-10T19:20:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]': finished 2024-09-10T19:20:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:36 smithi194 ceph-mon[32464]: osdmap e764: 8 total, 8 up, 8 in 2024-09-10T19:20:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:36 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/1945519470' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:20:36.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:36 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:20:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:36 smithi119 ceph-mon[29577]: pgmap v1254: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-09-10T19:20:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]': finished 2024-09-10T19:20:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:36 smithi119 ceph-mon[29577]: osdmap e764: 8 total, 8 up, 8 in 2024-09-10T19:20:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:36 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/1945519470' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:20:36.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:36 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:20:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:36 smithi119 ceph-mon[40674]: pgmap v1254: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-09-10T19:20:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]': finished 2024-09-10T19:20:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:36 smithi119 ceph-mon[40674]: osdmap e764: 8 total, 8 up, 8 in 2024-09-10T19:20:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:36 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/1945519470' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:20:36.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:36 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]: dispatch 2024-09-10T19:20:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:37 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]': finished 2024-09-10T19:20:37.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:37 smithi194 ceph-mon[32464]: osdmap e765: 8 total, 8 up, 8 in 2024-09-10T19:20:37.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:37 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]': finished 2024-09-10T19:20:37.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:37 smithi119 ceph-mon[29577]: osdmap e765: 8 total, 8 up, 8 in 2024-09-10T19:20:37.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:37 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "field": "max_objects", "val": "0"}]': finished 2024-09-10T19:20:37.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:37 smithi119 ceph-mon[40674]: osdmap e765: 8 total, 8 up, 8 in 2024-09-10T19:20:38.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:38 smithi194 ceph-mon[32464]: pgmap v1257: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:20:38.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:38 smithi119 ceph-mon[29577]: pgmap v1257: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:20:38.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:38 smithi119 ceph-mon[40674]: pgmap v1257: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:20:39.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:20:39 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:20:40.474 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:20:40 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:20:40] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:20:40.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:40 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:40.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:40 smithi194 ceph-mon[32464]: pgmap v1258: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2024-09-10T19:20:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:40 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:40.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:40 smithi119 ceph-mon[29577]: pgmap v1258: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2024-09-10T19:20:40.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:40 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:40.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:40 smithi119 ceph-mon[40674]: pgmap v1258: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2024-09-10T19:20:42.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:42 smithi194 ceph-mon[32464]: pgmap v1259: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-09-10T19:20:42.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:42 smithi119 ceph-mon[29577]: pgmap v1259: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-09-10T19:20:42.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:20:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:20:42] "GET /metrics HTTP/1.1" 200 228647 "" "Prometheus/2.43.0" 2024-09-10T19:20:42.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:42 smithi119 ceph-mon[40674]: pgmap v1259: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-09-10T19:20:44.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:44 smithi194 ceph-mon[32464]: pgmap v1260: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:44.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:44 smithi119 ceph-mon[29577]: pgmap v1260: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:44.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:44 smithi119 ceph-mon[40674]: pgmap v1260: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:46.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:46 smithi194 ceph-mon[32464]: pgmap v1261: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:20:46.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:46 smithi119 ceph-mon[29577]: pgmap v1261: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:20:46.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:46 smithi119 ceph-mon[40674]: pgmap v1261: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-09-10T19:20:47.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:47 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:47.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:47 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:47.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:47 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:48.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:48 smithi194 ceph-mon[32464]: pgmap v1262: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-10T19:20:48.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:48 smithi119 ceph-mon[29577]: pgmap v1262: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-10T19:20:48.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:48 smithi119 ceph-mon[40674]: pgmap v1262: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-09-10T19:20:49.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:20:49 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:20:50.494 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:20:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:20:50] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:20:50.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:50 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:50.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:50 smithi194 ceph-mon[32464]: pgmap v1263: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:50 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:50.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:50 smithi119 ceph-mon[29577]: pgmap v1263: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:50.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:50 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:50.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:50 smithi119 ceph-mon[40674]: pgmap v1263: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:52.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:52 smithi194 ceph-mon[32464]: pgmap v1264: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:20:52.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:52 smithi119 ceph-mon[29577]: pgmap v1264: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:20:52.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:52 smithi119 ceph-mon[40674]: pgmap v1264: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:20:52.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:20:52 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:20:52] "GET /metrics HTTP/1.1" 200 228643 "" "Prometheus/2.43.0" 2024-09-10T19:20:54.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:54 smithi194 ceph-mon[32464]: pgmap v1265: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:54 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:20:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:54 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:20:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:54 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:20:54.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:54 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:20:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:54 smithi119 ceph-mon[29577]: pgmap v1265: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:54 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:20:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:54 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:20:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:54 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:20:54.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:54 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:20:54.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:54 smithi119 ceph-mon[40674]: pgmap v1265: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:20:54.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:54 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:20:54.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:54 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:20:54.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:54 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:20:54.818 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:54 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' 2024-09-10T19:20:56.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:56 smithi194 ceph-mon[32464]: pgmap v1266: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:56.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:56 smithi119 ceph-mon[29577]: pgmap v1266: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:56.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:56 smithi119 ceph-mon[40674]: pgmap v1266: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:58.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:20:58 smithi194 ceph-mon[32464]: pgmap v1267: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:58.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:20:58 smithi119 ceph-mon[29577]: pgmap v1267: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:58.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:20:58 smithi119 ceph-mon[40674]: pgmap v1267: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:59.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:20:59 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:21:00.420 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:21:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:21:00] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:21:00.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:00 smithi194 ceph-mon[32464]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:00.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:00 smithi194 ceph-mon[32464]: pgmap v1268: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:21:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:00 smithi119 ceph-mon[29577]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:00.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:00 smithi119 ceph-mon[29577]: pgmap v1268: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:21:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:00 smithi119 ceph-mon[40674]: from='client.24410 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:00.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:00 smithi119 ceph-mon[40674]: pgmap v1268: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:21:02.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:02 smithi194 ceph-mon[32464]: pgmap v1269: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:21:02.769 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:02 smithi194 ceph-mon[32464]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:02.817 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:21:02 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:21:02] "GET /metrics HTTP/1.1" 200 228634 "" "Prometheus/2.43.0" 2024-09-10T19:21:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:02 smithi119 ceph-mon[29577]: pgmap v1269: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:21:02.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:02 smithi119 ceph-mon[29577]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:02.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:02 smithi119 ceph-mon[40674]: pgmap v1269: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-09-10T19:21:02.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:02 smithi119 ceph-mon[40674]: from='mgr.24452 172.21.15.119:0/3130723938' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:04.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:04 smithi194 ceph-mon[32464]: pgmap v1270: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:21:04.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:04 smithi119 ceph-mon[29577]: pgmap v1270: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:21:04.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:04 smithi119 ceph-mon[40674]: pgmap v1270: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:21:06.476 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd pool delete 791ac3f1-f414-442d-8950-49875e1f19bf 791ac3f1-f414-442d-8950-49875e1f19bf --yes-i-really-really-mean-it 2024-09-10T19:21:06.568 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.563+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/3272573950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed00fe950 msgr2=0x7f0ed00fedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:21:06.568 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.563+0000 7f0ed7c4f640 1 --2- 172.21.15.119:0/3272573950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed00fe950 0x7f0ed00fedc0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f0ec40048c0 tx=0x7f0ec4009da0 comp rx=0 tx=0).stop 2024-09-10T19:21:06.568 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.563+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/3272573950 shutdown_connections 2024-09-10T19:21:06.568 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.563+0000 7f0ed7c4f640 1 --2- 172.21.15.119:0/3272573950 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0ed00ff300 0x7f0ed010b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:06.568 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.563+0000 7f0ed7c4f640 1 --2- 172.21.15.119:0/3272573950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed00fe950 0x7f0ed00fedc0 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:06.568 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.563+0000 7f0ed7c4f640 1 --2- 172.21.15.119:0/3272573950 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ed0104970 0x7f0ed0104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:06.569 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.563+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/3272573950 >> 172.21.15.119:0/3272573950 conn(0x7f0ed00fa4f0 msgr2=0x7f0ed00fc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:21:06.569 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.563+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/3272573950 shutdown_connections 2024-09-10T19:21:06.569 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.563+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/3272573950 wait complete. 2024-09-10T19:21:06.569 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.564+0000 7f0ed7c4f640 1 Processor -- start 2024-09-10T19:21:06.569 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.564+0000 7f0ed7c4f640 1 -- start start 2024-09-10T19:21:06.569 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.564+0000 7f0ed7c4f640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed00fe950 0x7f0ed0114d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:21:06.569 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.564+0000 7f0ed7c4f640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ed00ff300 0x7f0ed01152c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:21:06.569 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.564+0000 7f0ed7c4f640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0ed0104970 0x7f0ed010ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:21:06.569 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.564+0000 7f0ed7c4f640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed010f5b0 con 0x7f0ed00fe950 2024-09-10T19:21:06.569 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.564+0000 7f0ed7c4f640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed010f720 con 0x7f0ed00ff300 2024-09-10T19:21:06.569 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.564+0000 7f0ed7c4f640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed010f8e0 con 0x7f0ed0104970 2024-09-10T19:21:06.570 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.564+0000 7f0ed59eb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed00fe950 0x7f0ed0114d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:21:06.570 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.564+0000 7f0ed59eb640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed00fe950 0x7f0ed0114d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37164/0 (socket says 172.21.15.119:37164) 2024-09-10T19:21:06.570 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.564+0000 7f0ed59eb640 1 -- 172.21.15.119:0/779945938 learned_addr learned my addr 172.21.15.119:0/779945938 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:21:06.570 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.564+0000 7f0ed61ec640 1 --2- 172.21.15.119:0/779945938 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0ed0104970 0x7f0ed010ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:21:06.570 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.564+0000 7f0ed51ea640 1 --2- 172.21.15.119:0/779945938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ed00ff300 0x7f0ed01152c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:21:06.570 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.565+0000 7f0ed59eb640 1 -- 172.21.15.119:0/779945938 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0ed0104970 msgr2=0x7f0ed010ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:21:06.570 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.565+0000 7f0ed59eb640 1 --2- 172.21.15.119:0/779945938 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0ed0104970 0x7f0ed010ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:06.570 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.565+0000 7f0ed59eb640 1 -- 172.21.15.119:0/779945938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ed00ff300 msgr2=0x7f0ed01152c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:21:06.570 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.565+0000 7f0ed59eb640 1 --2- 172.21.15.119:0/779945938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ed00ff300 0x7f0ed01152c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:06.570 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.565+0000 7f0ed59eb640 1 -- 172.21.15.119:0/779945938 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec00098b0 con 0x7f0ed00fe950 2024-09-10T19:21:06.570 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.565+0000 7f0ed51ea640 1 --2- 172.21.15.119:0/779945938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ed00ff300 0x7f0ed01152c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T19:21:06.571 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.565+0000 7f0ed59eb640 1 --2- 172.21.15.119:0/779945938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed00fe950 0x7f0ed0114d80 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800ebf0 tx=0x7f0ec800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:21:06.571 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.565+0000 7f0ece7fc640 1 -- 172.21.15.119:0/779945938 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8007c30 con 0x7f0ed00fe950 2024-09-10T19:21:06.571 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.565+0000 7f0ece7fc640 1 -- 172.21.15.119:0/779945938 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f0ec800f040 con 0x7f0ed00fe950 2024-09-10T19:21:06.571 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.565+0000 7f0ece7fc640 1 -- 172.21.15.119:0/779945938 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec80094f0 con 0x7f0ed00fe950 2024-09-10T19:21:06.571 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.565+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/779945938 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec4005040 con 0x7f0ed00fe950 2024-09-10T19:21:06.571 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.565+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/779945938 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ed011bbd0 con 0x7f0ed00fe950 2024-09-10T19:21:06.571 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.566+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/779945938 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ed0100090 con 0x7f0ed00fe950 2024-09-10T19:21:06.571 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.566+0000 7f0ece7fc640 1 -- 172.21.15.119:0/779945938 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f0ec8009860 con 0x7f0ed00fe950 2024-09-10T19:21:06.572 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.567+0000 7f0ece7fc640 1 --2- 172.21.15.119:0/779945938 >> v2:172.21.15.119:6800/3118961733 conn(0x7f0eb806ef80 0x7f0eb8071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:21:06.572 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.567+0000 7f0ece7fc640 1 -- 172.21.15.119:0/779945938 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(765..765 src has 1..765) v4 ==== 8243+0+0 (secure 0 0 0) 0x7f0ec808ea00 con 0x7f0ed00fe950 2024-09-10T19:21:06.572 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.567+0000 7f0ed51ea640 1 --2- 172.21.15.119:0/779945938 >> v2:172.21.15.119:6800/3118961733 conn(0x7f0eb806ef80 0x7f0eb8071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:21:06.573 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.567+0000 7f0ece7fc640 1 -- 172.21.15.119:0/779945938 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=766}) v3 -- 0x7f0eb807abe0 con 0x7f0ed00fe950 2024-09-10T19:21:06.573 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.568+0000 7f0ed51ea640 1 --2- 172.21.15.119:0/779945938 >> v2:172.21.15.119:6800/3118961733 conn(0x7f0eb806ef80 0x7f0eb8071430 secure :-1 s=READY pgs=3742 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0110640 tx=0x7f0ec4035040 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:21:06.573 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.571+0000 7f0ece7fc640 1 -- 172.21.15.119:0/779945938 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f0ec804e2a0 con 0x7f0ed00fe950 2024-09-10T19:21:06.730 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:06.728+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/779945938 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pool2": "791ac3f1-f414-442d-8950-49875e1f19bf", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0ed010de70 con 0x7f0ed00fe950 2024-09-10T19:21:06.768 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:06 smithi194 ceph-mon[32464]: pgmap v1271: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:21:06.816 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:06 smithi119 ceph-mon[29577]: pgmap v1271: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:21:06.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:06 smithi119 ceph-mon[40674]: pgmap v1271: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:21:07.524 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.522+0000 7f0ece7fc640 1 -- 172.21.15.119:0/779945938 <== mon.0 v2:172.21.15.119:3300/0 7 ==== osd_map(766..766 src has 1..766) v4 ==== 288+0+0 (secure 0 0 0) 0x7f0ec805af30 con 0x7f0ed00fe950 2024-09-10T19:21:07.524 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.522+0000 7f0ece7fc640 1 -- 172.21.15.119:0/779945938 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=767}) v3 -- 0x7f0eb807b510 con 0x7f0ed00fe950 2024-09-10T19:21:07.529 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.527+0000 7f0ece7fc640 1 -- 172.21.15.119:0/779945938 <== mon.0 v2:172.21.15.119:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pool2": "791ac3f1-f414-442d-8950-49875e1f19bf", "yes_i_really_really_mean_it": true}]=0 pool '791ac3f1-f414-442d-8950-49875e1f19bf' removed v766) v1 ==== 248+0+0 (secure 0 0 0) 0x7f0ec8019060 con 0x7f0ed00fe950 2024-09-10T19:21:07.628 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.627+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/779945938 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pool2": "791ac3f1-f414-442d-8950-49875e1f19bf", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0ed01104a0 con 0x7f0ed00fe950 2024-09-10T19:21:07.629 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.627+0000 7f0ece7fc640 1 -- 172.21.15.119:0/779945938 <== mon.0 v2:172.21.15.119:3300/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pool2": "791ac3f1-f414-442d-8950-49875e1f19bf", "yes_i_really_really_mean_it": true}]=0 pool '791ac3f1-f414-442d-8950-49875e1f19bf' does not exist v766) v1 ==== 255+0+0 (secure 0 0 0) 0x7f0ed01104a0 con 0x7f0ed00fe950 2024-09-10T19:21:07.629 INFO:tasks.workunit.client.0.smithi119.stderr:pool '791ac3f1-f414-442d-8950-49875e1f19bf' does not exist 2024-09-10T19:21:07.631 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.629+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/779945938 >> v2:172.21.15.119:6800/3118961733 conn(0x7f0eb806ef80 msgr2=0x7f0eb8071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:21:07.631 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.629+0000 7f0ed7c4f640 1 --2- 172.21.15.119:0/779945938 >> v2:172.21.15.119:6800/3118961733 conn(0x7f0eb806ef80 0x7f0eb8071430 secure :-1 s=READY pgs=3742 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0110640 tx=0x7f0ec4035040 comp rx=0 tx=0).stop 2024-09-10T19:21:07.631 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.629+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/779945938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed00fe950 msgr2=0x7f0ed0114d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:21:07.631 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.629+0000 7f0ed7c4f640 1 --2- 172.21.15.119:0/779945938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed00fe950 0x7f0ed0114d80 secure :-1 s=READY pgs=2474 cs=0 l=1 rev1=1 crypto rx=0x7f0ec800ebf0 tx=0x7f0ec800b040 comp rx=0 tx=0).stop 2024-09-10T19:21:07.631 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.629+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/779945938 shutdown_connections 2024-09-10T19:21:07.631 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.629+0000 7f0ed7c4f640 1 --2- 172.21.15.119:0/779945938 >> v2:172.21.15.119:6800/3118961733 conn(0x7f0eb806ef80 0x7f0eb8071430 unknown :-1 s=CLOSED pgs=3742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:07.631 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.629+0000 7f0ed7c4f640 1 --2- 172.21.15.119:0/779945938 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f0ed0104970 0x7f0ed010ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:07.632 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.629+0000 7f0ed7c4f640 1 --2- 172.21.15.119:0/779945938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ed00ff300 0x7f0ed01152c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:07.632 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.629+0000 7f0ed7c4f640 1 --2- 172.21.15.119:0/779945938 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ed00fe950 0x7f0ed0114d80 unknown :-1 s=CLOSED pgs=2474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:07.632 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.629+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/779945938 >> 172.21.15.119:0/779945938 conn(0x7f0ed00fa4f0 msgr2=0x7f0ed01024a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:21:07.632 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.629+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/779945938 shutdown_connections 2024-09-10T19:21:07.632 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.629+0000 7f0ed7c4f640 1 -- 172.21.15.119:0/779945938 wait complete. 2024-09-10T19:21:07.640 INFO:tasks.workunit.client.0.smithi119.stderr:+ ceph osd pool delete 0ceff89d-376b-4634-ba36-dca6344065fb 0ceff89d-376b-4634-ba36-dca6344065fb --yes-i-really-really-mean-it 2024-09-10T19:21:07.719 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.715+0000 7f4272e8b640 1 -- 172.21.15.119:0/3306030845 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f426c101f90 msgr2=0x7f426c0fb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:21:07.719 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.715+0000 7f4272e8b640 1 --2- 172.21.15.119:0/3306030845 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f426c101f90 0x7f426c0fb650 secure :-1 s=READY pgs=2978 cs=0 l=1 rev1=1 crypto rx=0x7f4258004890 tx=0x7f4258009da0 comp rx=0 tx=0).stop 2024-09-10T19:21:07.720 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.715+0000 7f4272e8b640 1 -- 172.21.15.119:0/3306030845 shutdown_connections 2024-09-10T19:21:07.720 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.715+0000 7f4272e8b640 1 --2- 172.21.15.119:0/3306030845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f426c0fbd30 0x7f426c1083d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:07.720 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.715+0000 7f4272e8b640 1 --2- 172.21.15.119:0/3306030845 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f426c101f90 0x7f426c0fb650 unknown :-1 s=CLOSED pgs=2978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:07.720 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.715+0000 7f4272e8b640 1 --2- 172.21.15.119:0/3306030845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f426c1015f0 0x7f426c1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:07.720 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.715+0000 7f4272e8b640 1 -- 172.21.15.119:0/3306030845 >> 172.21.15.119:0/3306030845 conn(0x7f426c0725d0 msgr2=0x7f426c0f9850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:21:07.721 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.715+0000 7f4272e8b640 1 -- 172.21.15.119:0/3306030845 shutdown_connections 2024-09-10T19:21:07.721 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.715+0000 7f4272e8b640 1 -- 172.21.15.119:0/3306030845 wait complete. 2024-09-10T19:21:07.721 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4272e8b640 1 Processor -- start 2024-09-10T19:21:07.721 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4272e8b640 1 -- start start 2024-09-10T19:21:07.721 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4272e8b640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f426c0fbd30 0x7f426c108c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:21:07.721 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4272e8b640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f426c1015f0 0x7f426c109140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:21:07.721 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4272e8b640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f426c101f90 0x7f426c109680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:21:07.721 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4272e8b640 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c11b3f0 con 0x7f426c101f90 2024-09-10T19:21:07.721 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4272e8b640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426c11b560 con 0x7f426c1015f0 2024-09-10T19:21:07.721 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4272e8b640 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f426c11b720 con 0x7f426c0fbd30 2024-09-10T19:21:07.721 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4270c27640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f426c0fbd30 0x7f426c108c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:21:07.722 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4270c27640 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f426c0fbd30 0x7f426c108c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:53550/0 (socket says 172.21.15.119:53550) 2024-09-10T19:21:07.722 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4271428640 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f426c101f90 0x7f426c109680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:21:07.722 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4270c27640 1 -- 172.21.15.119:0/3341757271 learned_addr learned my addr 172.21.15.119:0/3341757271 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-10T19:21:07.722 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f426bfff640 1 --2- 172.21.15.119:0/3341757271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f426c1015f0 0x7f426c109140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:21:07.722 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4270c27640 1 -- 172.21.15.119:0/3341757271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f426c1015f0 msgr2=0x7f426c109140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:21:07.722 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4270c27640 1 --2- 172.21.15.119:0/3341757271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f426c1015f0 0x7f426c109140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:07.722 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4270c27640 1 -- 172.21.15.119:0/3341757271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f426c101f90 msgr2=0x7f426c109680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:21:07.722 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4270c27640 1 --2- 172.21.15.119:0/3341757271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f426c101f90 0x7f426c109680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:07.722 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4270c27640 1 -- 172.21.15.119:0/3341757271 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42640098b0 con 0x7f426c0fbd30 2024-09-10T19:21:07.722 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.716+0000 7f4271428640 1 --2- 172.21.15.119:0/3341757271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f426c101f90 0x7f426c109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T19:21:07.722 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.717+0000 7f4270c27640 1 --2- 172.21.15.119:0/3341757271 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f426c0fbd30 0x7f426c108c00 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7f426000b810 tx=0x7f42600091d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:21:07.722 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.717+0000 7f42697fa640 1 -- 172.21.15.119:0/3341757271 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426000be60 con 0x7f426c0fbd30 2024-09-10T19:21:07.722 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.717+0000 7f42697fa640 1 -- 172.21.15.119:0/3341757271 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(24 keys) v1 ==== 1019+0+0 (secure 0 0 0) 0x7f426000cba0 con 0x7f426c0fbd30 2024-09-10T19:21:07.723 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.717+0000 7f4272e8b640 1 -- 172.21.15.119:0/3341757271 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4258005040 con 0x7f426c0fbd30 2024-09-10T19:21:07.723 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.717+0000 7f42697fa640 1 -- 172.21.15.119:0/3341757271 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4260015b60 con 0x7f426c0fbd30 2024-09-10T19:21:07.723 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.717+0000 7f4272e8b640 1 -- 172.21.15.119:0/3341757271 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426c11bcc0 con 0x7f426c0fbd30 2024-09-10T19:21:07.723 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.718+0000 7f4272e8b640 1 -- 172.21.15.119:0/3341757271 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426c0654c0 con 0x7f426c0fbd30 2024-09-10T19:21:07.724 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.718+0000 7f42697fa640 1 -- 172.21.15.119:0/3341757271 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 22) v1 ==== 90804+0+0 (secure 0 0 0) 0x7f4260012070 con 0x7f426c0fbd30 2024-09-10T19:21:07.724 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.719+0000 7f42697fa640 1 --2- 172.21.15.119:0/3341757271 >> v2:172.21.15.119:6800/3118961733 conn(0x7f425c06ef80 0x7f425c071430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T19:21:07.724 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.719+0000 7f42697fa640 1 -- 172.21.15.119:0/3341757271 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(766..766 src has 1..766) v4 ==== 7855+0+0 (secure 0 0 0) 0x7f426008e290 con 0x7f426c0fbd30 2024-09-10T19:21:07.724 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.719+0000 7f426bfff640 1 --2- 172.21.15.119:0/3341757271 >> v2:172.21.15.119:6800/3118961733 conn(0x7f425c06ef80 0x7f425c071430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T19:21:07.724 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.719+0000 7f42697fa640 1 -- 172.21.15.119:0/3341757271 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=767}) v3 -- 0x7f425c07a340 con 0x7f426c0fbd30 2024-09-10T19:21:07.724 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.719+0000 7f426bfff640 1 --2- 172.21.15.119:0/3341757271 >> v2:172.21.15.119:6800/3118961733 conn(0x7f425c06ef80 0x7f425c071430 secure :-1 s=READY pgs=3743 cs=0 l=1 rev1=1 crypto rx=0x7f4258004860 tx=0x7f4258009da0 comp rx=0 tx=0).ready entity=mgr.24452 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T19:21:07.724 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.723+0000 7f42697fa640 1 -- 172.21.15.119:0/3341757271 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176781 (secure 0 0 0) 0x7f426005ab10 con 0x7f426c0fbd30 2024-09-10T19:21:07.817 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:07 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/779945938' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pool2": "791ac3f1-f414-442d-8950-49875e1f19bf", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:07.817 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:07 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/779945938' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pool2": "791ac3f1-f414-442d-8950-49875e1f19bf", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:07.880 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:07.878+0000 7f4272e8b640 1 -- 172.21.15.119:0/3341757271 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f426c10bbc0 con 0x7f426c0fbd30 2024-09-10T19:21:08.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:07 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/779945938' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pool2": "791ac3f1-f414-442d-8950-49875e1f19bf", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:08.531 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.529+0000 7f42697fa640 1 -- 172.21.15.119:0/3341757271 <== mon.2 v2:172.21.15.119:3301/0 7 ==== osd_map(767..767 src has 1..767) v4 ==== 288+0+0 (secure 0 0 0) 0x7f426005a850 con 0x7f426c0fbd30 2024-09-10T19:21:08.531 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.529+0000 7f42697fa640 1 -- 172.21.15.119:0/3341757271 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=768}) v3 -- 0x7f425c07ac30 con 0x7f426c0fbd30 2024-09-10T19:21:08.536 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.535+0000 7f42697fa640 1 -- 172.21.15.119:0/3341757271 <== mon.2 v2:172.21.15.119:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]=0 pool '0ceff89d-376b-4634-ba36-dca6344065fb' removed v767) v1 ==== 248+0+0 (secure 0 0 0) 0x7f426005a4b0 con 0x7f426c0fbd30 2024-09-10T19:21:08.633 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.632+0000 7f4272e8b640 1 -- 172.21.15.119:0/3341757271 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f426c10a0c0 con 0x7f426c0fbd30 2024-09-10T19:21:08.634 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.633+0000 7f42697fa640 1 -- 172.21.15.119:0/3341757271 <== mon.2 v2:172.21.15.119:3301/0 9 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]=0 pool '0ceff89d-376b-4634-ba36-dca6344065fb' does not exist v767) v1 ==== 255+0+0 (secure 0 0 0) 0x7f426005a690 con 0x7f426c0fbd30 2024-09-10T19:21:08.634 INFO:tasks.workunit.client.0.smithi119.stderr:pool '0ceff89d-376b-4634-ba36-dca6344065fb' does not exist 2024-09-10T19:21:08.637 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.635+0000 7f4272e8b640 1 -- 172.21.15.119:0/3341757271 >> v2:172.21.15.119:6800/3118961733 conn(0x7f425c06ef80 msgr2=0x7f425c071430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:21:08.637 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.635+0000 7f4272e8b640 1 --2- 172.21.15.119:0/3341757271 >> v2:172.21.15.119:6800/3118961733 conn(0x7f425c06ef80 0x7f425c071430 secure :-1 s=READY pgs=3743 cs=0 l=1 rev1=1 crypto rx=0x7f4258004860 tx=0x7f4258009da0 comp rx=0 tx=0).stop 2024-09-10T19:21:08.637 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.635+0000 7f4272e8b640 1 -- 172.21.15.119:0/3341757271 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f426c0fbd30 msgr2=0x7f426c108c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T19:21:08.637 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.635+0000 7f4272e8b640 1 --2- 172.21.15.119:0/3341757271 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f426c0fbd30 0x7f426c108c00 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7f426000b810 tx=0x7f42600091d0 comp rx=0 tx=0).stop 2024-09-10T19:21:08.637 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.635+0000 7f4272e8b640 1 -- 172.21.15.119:0/3341757271 shutdown_connections 2024-09-10T19:21:08.637 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.635+0000 7f4272e8b640 1 --2- 172.21.15.119:0/3341757271 >> v2:172.21.15.119:6800/3118961733 conn(0x7f425c06ef80 0x7f425c071430 unknown :-1 s=CLOSED pgs=3743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:08.637 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.635+0000 7f4272e8b640 1 --2- 172.21.15.119:0/3341757271 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f426c101f90 0x7f426c109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:08.637 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.635+0000 7f4272e8b640 1 --2- 172.21.15.119:0/3341757271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f426c1015f0 0x7f426c109140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:08.637 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.635+0000 7f4272e8b640 1 --2- 172.21.15.119:0/3341757271 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f426c0fbd30 0x7f426c108c00 unknown :-1 s=CLOSED pgs=2979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T19:21:08.637 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.635+0000 7f4272e8b640 1 -- 172.21.15.119:0/3341757271 >> 172.21.15.119:0/3341757271 conn(0x7f426c0725d0 msgr2=0x7f426c0ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T19:21:08.637 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.635+0000 7f4272e8b640 1 -- 172.21.15.119:0/3341757271 shutdown_connections 2024-09-10T19:21:08.637 INFO:tasks.workunit.client.0.smithi119.stderr:2024-09-10T19:21:08.635+0000 7f4272e8b640 1 -- 172.21.15.119:0/3341757271 wait complete. 2024-09-10T19:21:08.645 INFO:tasks.workunit.client.0.smithi119.stderr:+ echo OK 2024-09-10T19:21:08.646 INFO:tasks.workunit.client.0.smithi119.stdout:OK 2024-09-10T19:21:08.647 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T19:21:08.647 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T19:21:08.714 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2024-09-10T19:21:08.714 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-10T19:21:08.776 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:08 smithi119 ceph-mon[29577]: pgmap v1272: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:21:08.776 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/779945938' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pool2": "791ac3f1-f414-442d-8950-49875e1f19bf", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:21:08.776 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:08 smithi119 ceph-mon[29577]: osdmap e766: 8 total, 8 up, 8 in 2024-09-10T19:21:08.777 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/779945938' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pool2": "791ac3f1-f414-442d-8950-49875e1f19bf", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:08.777 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:08 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3341757271' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:08.777 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:08 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:08.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:08 smithi119 ceph-mon[40674]: pgmap v1272: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:21:08.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/779945938' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pool2": "791ac3f1-f414-442d-8950-49875e1f19bf", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:21:08.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:08 smithi119 ceph-mon[40674]: osdmap e766: 8 total, 8 up, 8 in 2024-09-10T19:21:08.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/779945938' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pool2": "791ac3f1-f414-442d-8950-49875e1f19bf", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:08.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:08 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3341757271' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:08.777 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:08 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:09.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:08 smithi194 ceph-mon[32464]: pgmap v1272: 188 pgs: 188 active+clean; 894 KiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:21:09.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/779945938' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pool2": "791ac3f1-f414-442d-8950-49875e1f19bf", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:21:09.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:08 smithi194 ceph-mon[32464]: osdmap e766: 8 total, 8 up, 8 in 2024-09-10T19:21:09.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/779945938' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "791ac3f1-f414-442d-8950-49875e1f19bf", "pool2": "791ac3f1-f414-442d-8950-49875e1f19bf", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:09.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:08 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3341757271' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:09.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:08 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:09.032 DEBUG:teuthology.parallel:result is None 2024-09-10T19:21:09.032 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T19:21:09.097 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T19:21:09.098 DEBUG:teuthology.orchestra.run.smithi119:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-10T19:21:09.157 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T19:21:09.158 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-10T19:21:09.167 INFO:tasks.cephadm:Teardown begin 2024-09-10T19:21:09.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:21:09.222 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:21:09.256 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-10T19:21:09.256 DEBUG:teuthology.orchestra.run.smithi119:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-10T19:21:09.277 DEBUG:teuthology.orchestra.run.smithi194:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-10T19:21:09.313 INFO:tasks.cephadm:Stopping all daemons... 2024-09-10T19:21:09.313 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-10T19:21:09.313 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.a 2024-09-10T19:21:09.518 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:21:09 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug there is no tcmu-runner data available 2024-09-10T19:21:09.632 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:09 smithi119 systemd[1]: Stopping Ceph mon.a for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:21:09.632 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-mon[29577]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T19:21:09.632 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:21:09.632 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-mon[29577]: osdmap e767: 8 total, 8 up, 8 in 2024-09-10T19:21:09.632 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-mon[29577]: from='client.? 172.21.15.119:0/3341757271' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:09.632 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-mon[29577]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:09.633 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-mon[40674]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T19:21:09.633 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:21:09.633 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-mon[40674]: osdmap e767: 8 total, 8 up, 8 in 2024-09-10T19:21:09.633 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-mon[40674]: from='client.? 172.21.15.119:0/3341757271' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:09.633 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-mon[40674]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:09.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a[29554]: 2024-09-10T19:21:09.629+0000 7f52de710640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:21:09.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a[29554]: 2024-09-10T19:21:09.629+0000 7f52de710640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-10T19:21:09.964 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:09 smithi119 podman[149597]: 2024-09-10 19:21:09.661097392 +0000 UTC m=+0.163474927 container died f867ead4cf34929f3d407e5b032218276626003d1e36c4f22d3c814200cb81cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T19:21:09.965 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:21:09] ENGINE Bus STOPPING 2024-09-10T19:21:09.965 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:21:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T19:21:09.965 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:21:09] ENGINE Bus STOPPED 2024-09-10T19:21:09.965 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:21:09] ENGINE Bus STARTING 2024-09-10T19:21:09.965 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:21:09] ENGINE Serving on http://:::9283 2024-09-10T19:21:09.965 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:21:09 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: [10/Sep/2024:19:21:09] ENGINE Bus STARTED 2024-09-10T19:21:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:09 smithi194 ceph-mon[32464]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-09-10T19:21:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:21:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:09 smithi194 ceph-mon[32464]: osdmap e767: 8 total, 8 up, 8 in 2024-09-10T19:21:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:09 smithi194 ceph-mon[32464]: from='client.? 172.21.15.119:0/3341757271' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:10.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:09 smithi194 ceph-mon[32464]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "0ceff89d-376b-4634-ba36-dca6344065fb", "pool2": "0ceff89d-376b-4634-ba36-dca6344065fb", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:21:10.316 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:10 smithi119 podman[149597]: 2024-09-10 19:21:10.162567508 +0000 UTC m=+0.664945085 container cleanup f867ead4cf34929f3d407e5b032218276626003d1e36c4f22d3c814200cb81cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T19:21:10.317 INFO:journalctl@ceph.mon.a.smithi119.stdout:Sep 10 19:21:10 smithi119 bash[149597]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-a 2024-09-10T19:21:10.518 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:21:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x[34550]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:21:10] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:21:10.587 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.a.service' 2024-09-10T19:21:10.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:21:10.626 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-10T19:21:10.626 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-10T19:21:10.626 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.c 2024-09-10T19:21:10.953 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:10 smithi119 systemd[1]: Stopping Ceph mon.c for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:21:11.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:10 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-c[40651]: 2024-09-10T19:21:10.951+0000 7f75a49ba640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:21:11.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:10 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-c[40651]: 2024-09-10T19:21:10.951+0000 7f75a49ba640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-10T19:21:11.317 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:11 smithi119 podman[149740]: 2024-09-10 19:21:11.034342506 +0000 UTC m=+0.177967286 container died d7b0fe23492836672bccf895fcda7ee533c3817bfc0d209658d81cb4d519ce4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:21:11.678 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:11 smithi119 podman[149740]: 2024-09-10 19:21:11.425756967 +0000 UTC m=+0.569381788 container cleanup d7b0fe23492836672bccf895fcda7ee533c3817bfc0d209658d81cb4d519ce4f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-c, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-10T19:21:11.678 INFO:journalctl@ceph.mon.c.smithi119.stdout:Sep 10 19:21:11 smithi119 bash[149740]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-c 2024-09-10T19:21:11.835 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.c.service' 2024-09-10T19:21:11.869 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:21:11.869 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-10T19:21:11.869 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-10T19:21:11.870 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.b 2024-09-10T19:21:12.213 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:11 smithi194 systemd[1]: Stopping Ceph mon.b for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:21:12.477 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-b[32441]: 2024-09-10T19:21:12.211+0000 7f5de38e0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:21:12.477 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-b[32441]: 2024-09-10T19:21:12.211+0000 7f5de38e0640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-10T19:21:12.736 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:12 smithi194 podman[71650]: 2024-09-10 19:21:12.476718086 +0000 UTC m=+0.406022470 container died 8762862dea5c36023485aa02040f92a7c6ce35c0d06943310900f8dd840dbbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-b, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-10T19:21:13.018 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:12 smithi194 podman[71650]: 2024-09-10 19:21:12.917780481 +0000 UTC m=+0.847084865 container cleanup 8762862dea5c36023485aa02040f92a7c6ce35c0d06943310900f8dd840dbbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-b, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-10T19:21:13.019 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:12 smithi194 bash[71650]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-b 2024-09-10T19:21:13.067 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:21:12 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: ::ffff:172.21.15.194 - - [10/Sep/2024:19:21:12] "GET /metrics HTTP/1.1" 200 228634 "" "Prometheus/2.43.0" 2024-09-10T19:21:13.350 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.b.service' 2024-09-10T19:21:13.372 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:13 smithi194 podman[71667]: 2024-09-10 19:21:13.110930718 +0000 UTC m=+0.627118107 container remove 8762862dea5c36023485aa02040f92a7c6ce35c0d06943310900f8dd840dbbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mon-b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:21:13.372 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:13 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.b.service: Deactivated successfully. 2024-09-10T19:21:13.372 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:13 smithi194 systemd[1]: Stopped Ceph mon.b for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:21:13.373 INFO:journalctl@ceph.mon.b.smithi194.stdout:Sep 10 19:21:13 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mon.b.service: Consumed 23.021s CPU time. 2024-09-10T19:21:13.426 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:21:13.427 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-10T19:21:13.427 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-10T19:21:13.427 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mgr.y 2024-09-10T19:21:13.816 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:21:13 smithi119 systemd[1]: Stopping Ceph mgr.y for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:21:14.150 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:21:13 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:21:13.814+0000 7f07beabe640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:21:14.150 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:21:13 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y[29827]: 2024-09-10T19:21:13.814+0000 7f07beabe640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-10T19:21:14.150 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:21:13 smithi119 podman[149870]: 2024-09-10 19:21:13.846621301 +0000 UTC m=+0.219435100 container died 375fba9bae7df0355a5c36b86f8af75d81bdec2895c374ca4d7fae7201822fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2) 2024-09-10T19:21:14.534 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:21:14 smithi119 podman[149870]: 2024-09-10 19:21:14.222868933 +0000 UTC m=+0.595682784 container cleanup 375fba9bae7df0355a5c36b86f8af75d81bdec2895c374ca4d7fae7201822fdb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-10T19:21:14.534 INFO:journalctl@ceph.mgr.y.smithi119.stdout:Sep 10 19:21:14 smithi119 bash[149870]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-y 2024-09-10T19:21:14.694 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mgr.y.service' 2024-09-10T19:21:14.729 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:21:14.729 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-10T19:21:14.729 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-10T19:21:14.729 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mgr.x 2024-09-10T19:21:15.018 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:21:14 smithi194 systemd[1]: Stopping Ceph mgr.x for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:21:15.299 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:21:15 smithi194 podman[71782]: 2024-09-10 19:21:15.037383682 +0000 UTC m=+0.116936391 container died 1f9b2bbe1ba1d2ecf98bdebaaacfb3798d667e055fc895bc436f38baa2dbcc26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:21:15.574 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:21:15 smithi194 podman[71782]: 2024-09-10 19:21:15.380692362 +0000 UTC m=+0.460245070 container cleanup 1f9b2bbe1ba1d2ecf98bdebaaacfb3798d667e055fc895bc436f38baa2dbcc26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-10T19:21:15.575 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:21:15 smithi194 bash[71782]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x 2024-09-10T19:21:15.812 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mgr.x.service' 2024-09-10T19:21:15.834 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:21:15 smithi194 podman[71799]: 2024-09-10 19:21:15.573704185 +0000 UTC m=+0.531452848 container remove 1f9b2bbe1ba1d2ecf98bdebaaacfb3798d667e055fc895bc436f38baa2dbcc26 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-mgr-x, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-10T19:21:15.834 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:21:15 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mgr.x.service: Main process exited, code=exited, status=143/n/a 2024-09-10T19:21:15.834 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:21:15 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mgr.x.service: Failed with result 'exit-code'. 2024-09-10T19:21:15.834 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:21:15 smithi194 systemd[1]: Stopped Ceph mgr.x for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:21:15.835 INFO:journalctl@ceph.mgr.x.smithi194.stdout:Sep 10 19:21:15 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@mgr.x.service: Consumed 11.080s CPU time. 2024-09-10T19:21:15.848 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:21:15.848 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-10T19:21:15.848 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-10T19:21:15.849 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.0 2024-09-10T19:21:16.316 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:21:15 smithi119 systemd[1]: Stopping Ceph osd.0 for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:21:16.817 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:21:16 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0[44998]: 2024-09-10T19:21:16.420+0000 7f95596fe640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:21:16.817 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:21:16 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0[44998]: 2024-09-10T19:21:16.420+0000 7f95596fe640 -1 osd.0 767 *** Got signal Terminated *** 2024-09-10T19:21:16.817 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:21:16 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0[44998]: 2024-09-10T19:21:16.420+0000 7f95596fe640 -1 osd.0 767 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:21:22.067 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:21:21 smithi119 podman[150003]: 2024-09-10 19:21:21.749875163 +0000 UTC m=+5.707718001 container died 18091707a930e58b27d5904981e7c4553e1e941eafd97b4b10bc234c578b92ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:21:22.413 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:21:22 smithi119 podman[150003]: 2024-09-10 19:21:22.168912548 +0000 UTC m=+6.126755385 container cleanup 18091707a930e58b27d5904981e7c4553e1e941eafd97b4b10bc234c578b92ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:21:22.413 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:21:22 smithi119 bash[150003]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0 2024-09-10T19:21:22.812 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:21:22 smithi119 podman[150024]: 2024-09-10 19:21:22.41240349 +0000 UTC m=+0.655303049 container remove 18091707a930e58b27d5904981e7c4553e1e941eafd97b4b10bc234c578b92ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True) 2024-09-10T19:21:23.067 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:21:22 smithi119 podman[150120]: 2024-09-10 19:21:22.714239809 +0000 UTC m=+0.015808683 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:21:23.067 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:21:22 smithi119 podman[150120]: 2024-09-10 19:21:22.813107448 +0000 UTC m=+0.114676311 container create 9560b9d8a70f3ad385089a1f3e1eab06187c5976238c3b9f97600dab70fbf200 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-10T19:21:23.067 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:21:23 smithi119 podman[150120]: 2024-09-10 19:21:23.065884621 +0000 UTC m=+0.367453497 container init 9560b9d8a70f3ad385089a1f3e1eab06187c5976238c3b9f97600dab70fbf200 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-10T19:21:23.317 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:21:23 smithi119 podman[150120]: 2024-09-10 19:21:23.070638241 +0000 UTC m=+0.372207106 container start 9560b9d8a70f3ad385089a1f3e1eab06187c5976238c3b9f97600dab70fbf200 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-09-10T19:21:23.317 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:21:23 smithi119 podman[150120]: 2024-09-10 19:21:23.165616346 +0000 UTC m=+0.467185220 container attach 9560b9d8a70f3ad385089a1f3e1eab06187c5976238c3b9f97600dab70fbf200 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-10T19:21:23.317 INFO:journalctl@ceph.osd.0.smithi119.stdout:Sep 10 19:21:23 smithi119 podman[150120]: 2024-09-10 19:21:23.245320059 +0000 UTC m=+0.546888924 container died 9560b9d8a70f3ad385089a1f3e1eab06187c5976238c3b9f97600dab70fbf200 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-10T19:21:23.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:23 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:23.569Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:21:23.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:23 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:23.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:21:23.867 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.0.service' 2024-09-10T19:21:23.906 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:21:23.907 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-10T19:21:23.907 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-10T19:21:23.907 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.1 2024-09-10T19:21:24.233 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:21:23 smithi119 systemd[1]: Stopping Ceph osd.1 for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:21:24.510 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:21:24 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1[50994]: 2024-09-10T19:21:24.231+0000 7f757bd8f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:21:24.510 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:21:24 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1[50994]: 2024-09-10T19:21:24.231+0000 7f757bd8f640 -1 osd.1 767 *** Got signal Terminated *** 2024-09-10T19:21:24.510 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:21:24 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1[50994]: 2024-09-10T19:21:24.231+0000 7f757bd8f640 -1 osd.1 767 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:21:29.908 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:21:29 smithi119 podman[150261]: 2024-09-10 19:21:29.635592253 +0000 UTC m=+5.504808853 container died d4a4b62f44c0ad883163edb099e776e5294399a6c6bc861bcd50f29e7e6b87b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=) 2024-09-10T19:21:30.274 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:21:29 smithi119 podman[150261]: 2024-09-10 19:21:29.980478264 +0000 UTC m=+5.849694854 container cleanup d4a4b62f44c0ad883163edb099e776e5294399a6c6bc861bcd50f29e7e6b87b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-10T19:21:30.275 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:21:29 smithi119 bash[150261]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1 2024-09-10T19:21:30.275 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:30 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:30.148Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:21:30.275 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:30 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:30.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:21:30.566 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:21:30 smithi119 podman[150281]: 2024-09-10 19:21:30.273351573 +0000 UTC m=+0.636193881 container remove d4a4b62f44c0ad883163edb099e776e5294399a6c6bc861bcd50f29e7e6b87b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-10T19:21:30.928 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:21:30 smithi119 podman[150376]: 2024-09-10 19:21:30.495158296 +0000 UTC m=+0.015538878 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:21:30.928 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:21:30 smithi119 podman[150376]: 2024-09-10 19:21:30.615903689 +0000 UTC m=+0.136284259 container create b3f80d470afa20825b310c4c173ae2daf4569e64548e4f0929ce41df05275104 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-10T19:21:30.928 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:21:30 smithi119 podman[150376]: 2024-09-10 19:21:30.834936245 +0000 UTC m=+0.355316815 container init b3f80d470afa20825b310c4c173ae2daf4569e64548e4f0929ce41df05275104 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-10T19:21:30.928 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:21:30 smithi119 podman[150376]: 2024-09-10 19:21:30.839746285 +0000 UTC m=+0.360126874 container start b3f80d470afa20825b310c4c173ae2daf4569e64548e4f0929ce41df05275104 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1-deactivate, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-10T19:21:30.928 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:21:30 smithi119 podman[150376]: 2024-09-10 19:21:30.926766835 +0000 UTC m=+0.447147409 container attach b3f80d470afa20825b310c4c173ae2daf4569e64548e4f0929ce41df05275104 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-10T19:21:31.305 INFO:journalctl@ceph.osd.1.smithi119.stdout:Sep 10 19:21:31 smithi119 podman[150376]: 2024-09-10 19:21:31.012208135 +0000 UTC m=+0.532588716 container died b3f80d470afa20825b310c4c173ae2daf4569e64548e4f0929ce41df05275104 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-10T19:21:31.609 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.1.service' 2024-09-10T19:21:31.647 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:21:31.648 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-10T19:21:31.648 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-10T19:21:31.648 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.2 2024-09-10T19:21:31.969 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:21:31 smithi119 systemd[1]: Stopping Ceph osd.2 for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:21:32.317 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:21:31 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2[56942]: 2024-09-10T19:21:31.967+0000 7f6956087640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:21:32.317 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:21:31 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2[56942]: 2024-09-10T19:21:31.967+0000 7f6956087640 -1 osd.2 767 *** Got signal Terminated *** 2024-09-10T19:21:32.317 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:21:31 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2[56942]: 2024-09-10T19:21:31.967+0000 7f6956087640 -1 osd.2 767 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:21:34.007 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:33.569Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:21:34.008 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:33.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:21:34.008 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:33 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:33.570Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:21:37.810 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:21:37 smithi119 podman[150504]: 2024-09-10 19:21:37.528976759 +0000 UTC m=+5.665991610 container died 3ef571724a993336b62fe2a27ab32a00b0a4b0ee050a4c03220f222ae959db31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:21:38.067 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:21:37 smithi119 podman[150504]: 2024-09-10 19:21:37.890923787 +0000 UTC m=+6.027938638 container cleanup 3ef571724a993336b62fe2a27ab32a00b0a4b0ee050a4c03220f222ae959db31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:21:38.067 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:21:37 smithi119 bash[150504]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2 2024-09-10T19:21:38.515 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:21:38 smithi119 podman[150525]: 2024-09-10 19:21:38.166939519 +0000 UTC m=+0.640586570 container remove 3ef571724a993336b62fe2a27ab32a00b0a4b0ee050a4c03220f222ae959db31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=) 2024-09-10T19:21:38.768 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:21:38 smithi119 podman[150623]: 2024-09-10 19:21:38.417178915 +0000 UTC m=+0.015645615 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:21:38.768 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:21:38 smithi119 podman[150623]: 2024-09-10 19:21:38.534604706 +0000 UTC m=+0.133071435 container create e82f8fe7948ac3f5ba5ff7ef54671609fce6462ed5fc9ade8f454dee938d2b5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:21:38.768 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:21:38 smithi119 podman[150623]: 2024-09-10 19:21:38.762512258 +0000 UTC m=+0.360978953 container init e82f8fe7948ac3f5ba5ff7ef54671609fce6462ed5fc9ade8f454dee938d2b5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-10T19:21:38.768 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:21:38 smithi119 podman[150623]: 2024-09-10 19:21:38.767310384 +0000 UTC m=+0.365777072 container start e82f8fe7948ac3f5ba5ff7ef54671609fce6462ed5fc9ade8f454dee938d2b5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2) 2024-09-10T19:21:39.066 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:21:38 smithi119 podman[150623]: 2024-09-10 19:21:38.87033536 +0000 UTC m=+0.468802072 container attach e82f8fe7948ac3f5ba5ff7ef54671609fce6462ed5fc9ade8f454dee938d2b5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:21:39.067 INFO:journalctl@ceph.osd.2.smithi119.stdout:Sep 10 19:21:38 smithi119 podman[150623]: 2024-09-10 19:21:38.950317802 +0000 UTC m=+0.548784499 container died e82f8fe7948ac3f5ba5ff7ef54671609fce6462ed5fc9ade8f454dee938d2b5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True) 2024-09-10T19:21:39.754 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.2.service' 2024-09-10T19:21:39.793 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:21:39.793 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-10T19:21:39.793 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-10T19:21:39.794 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.3 2024-09-10T19:21:40.066 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:39 smithi119 systemd[1]: Stopping Ceph osd.3 for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:21:40.569 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:40 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3[63296]: 2024-09-10T19:21:40.119+0000 7fb6ba600640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:21:40.569 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:40 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3[63296]: 2024-09-10T19:21:40.119+0000 7fb6ba600640 -1 osd.3 767 *** Got signal Terminated *** 2024-09-10T19:21:40.569 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:40 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3[63296]: 2024-09-10T19:21:40.119+0000 7fb6ba600640 -1 osd.3 767 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:21:40.569 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:40 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:40.149Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:21:40.569 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:40 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:40.149Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:21:40.570 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:40 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:40.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:21:42.799 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:42 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:42.518+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:42.816 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:42 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3[63296]: 2024-09-10T19:21:42.519+0000 7fb6b6c18640 -1 osd.3 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:17.284524+0000 front 2024-09-10T19:21:17.284506+0000 (oldest deadline 2024-09-10T19:21:41.984648+0000) 2024-09-10T19:21:43.767 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:43 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:43.473+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:43.817 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:43 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3[63296]: 2024-09-10T19:21:43.518+0000 7fb6b6c18640 -1 osd.3 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:17.284524+0000 front 2024-09-10T19:21:17.284506+0000 (oldest deadline 2024-09-10T19:21:41.984648+0000) 2024-09-10T19:21:43.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:43 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:43.570Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:21:43.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:43 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:43.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:21:43.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:43 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:43.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:21:44.018 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:44 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:21:44.014+0000 7f08964d4640 -1 osd.4 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:17.905939+0000 front 2024-09-10T19:21:17.905834+0000 (oldest deadline 2024-09-10T19:21:43.805759+0000) 2024-09-10T19:21:44.769 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:44 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:44.488+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:44.769 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:44 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:44.725+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:44.817 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:44 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3[63296]: 2024-09-10T19:21:44.560+0000 7fb6b6c18640 -1 osd.3 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:17.284524+0000 front 2024-09-10T19:21:17.284506+0000 (oldest deadline 2024-09-10T19:21:41.984648+0000) 2024-09-10T19:21:45.268 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:44 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:21:44.984+0000 7f08964d4640 -1 osd.4 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:17.905939+0000 front 2024-09-10T19:21:17.905834+0000 (oldest deadline 2024-09-10T19:21:43.805759+0000) 2024-09-10T19:21:45.709 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:45 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:45.492+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:45.709 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:45 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:45.272+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:45.928 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:45 smithi119 podman[150751]: 2024-09-10 19:21:45.655958914 +0000 UTC m=+5.646807323 container died 7a9cf723c9072dd34f3baa1ad0f1f856c2e27bae9a81b81e03fcafbe93edf977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:21:46.018 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:45 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:45.707+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:46.203 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:46 smithi119 podman[150751]: 2024-09-10 19:21:46.001454654 +0000 UTC m=+5.992303063 container cleanup 7a9cf723c9072dd34f3baa1ad0f1f856c2e27bae9a81b81e03fcafbe93edf977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True) 2024-09-10T19:21:46.204 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:46 smithi119 bash[150751]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3 2024-09-10T19:21:46.446 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:46 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:21:46.017+0000 7f08964d4640 -1 osd.4 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:17.905939+0000 front 2024-09-10T19:21:17.905834+0000 (oldest deadline 2024-09-10T19:21:43.805759+0000) 2024-09-10T19:21:46.447 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:46 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:46.241+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:46.566 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:46 smithi119 podman[150771]: 2024-09-10 19:21:46.202560184 +0000 UTC m=+0.542220090 container remove 7a9cf723c9072dd34f3baa1ad0f1f856c2e27bae9a81b81e03fcafbe93edf977 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-10T19:21:46.735 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:46 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:46.444+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:46.873 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:46 smithi119 podman[150867]: 2024-09-10 19:21:46.603262216 +0000 UTC m=+0.103418528 container create e5195ef7d36a6e7797bc4b6f00267acd170fce4541334af4d44cc5b5b666379f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:21:46.873 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:46 smithi119 podman[150867]: 2024-09-10 19:21:46.51555809 +0000 UTC m=+0.015714408 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:21:46.873 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:46 smithi119 podman[150867]: 2024-09-10 19:21:46.789200385 +0000 UTC m=+0.289356699 container init e5195ef7d36a6e7797bc4b6f00267acd170fce4541334af4d44cc5b5b666379f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:21:46.873 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:46 smithi119 podman[150867]: 2024-09-10 19:21:46.794194808 +0000 UTC m=+0.294351127 container start e5195ef7d36a6e7797bc4b6f00267acd170fce4541334af4d44cc5b5b666379f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3-deactivate, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-10T19:21:46.873 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:46 smithi119 podman[150867]: 2024-09-10 19:21:46.872122985 +0000 UTC m=+0.372279314 container attach e5195ef7d36a6e7797bc4b6f00267acd170fce4541334af4d44cc5b5b666379f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD) 2024-09-10T19:21:47.018 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:46 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:46.733+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:47.251 INFO:journalctl@ceph.osd.3.smithi119.stdout:Sep 10 19:21:46 smithi119 podman[150867]: 2024-09-10 19:21:46.969614456 +0000 UTC m=+0.469770769 container died e5195ef7d36a6e7797bc4b6f00267acd170fce4541334af4d44cc5b5b666379f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-3-deactivate, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:21:47.469 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:47 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:21:47.051+0000 7f08964d4640 -1 osd.4 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:17.905939+0000 front 2024-09-10T19:21:17.905834+0000 (oldest deadline 2024-09-10T19:21:43.805759+0000) 2024-09-10T19:21:47.470 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:47 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:47.216+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:47.518 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.3.service' 2024-09-10T19:21:47.556 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:21:47.557 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-10T19:21:47.557 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-10T19:21:47.557 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.4 2024-09-10T19:21:47.742 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:47 smithi194 systemd[1]: Stopping Ceph osd.4 for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:21:47.742 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:47 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:47.467+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:48.018 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:47 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:47.740+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:48.019 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:47 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:21:47.866+0000 7f08986b9640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:21:48.019 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:47 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:21:47.866+0000 7f08986b9640 -1 osd.4 767 *** Got signal Terminated *** 2024-09-10T19:21:48.019 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:47 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:21:47.866+0000 7f08986b9640 -1 osd.4 767 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:21:48.445 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:21:48.076+0000 7f08964d4640 -1 osd.4 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:17.905939+0000 front 2024-09-10T19:21:17.905834+0000 (oldest deadline 2024-09-10T19:21:43.805759+0000) 2024-09-10T19:21:48.445 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:48.220+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:48.768 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:48.443+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:49.113 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:48 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:48.766+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:49.470 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:49 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:21:49.111+0000 7f08964d4640 -1 osd.4 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:17.905939+0000 front 2024-09-10T19:21:17.905834+0000 (oldest deadline 2024-09-10T19:21:43.805759+0000) 2024-09-10T19:21:49.470 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:49 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:49.239+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:49.768 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:49 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:49.468+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:50.082 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:49 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:49.816+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:50.490 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:21:50.080+0000 7f08964d4640 -1 osd.4 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:17.905939+0000 front 2024-09-10T19:21:17.905834+0000 (oldest deadline 2024-09-10T19:21:43.805759+0000) 2024-09-10T19:21:50.491 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:50.256+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:50.567 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:50 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:50.149Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:21:50.567 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:50 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:50.150Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:21:50.567 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:50 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:50.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:21:50.769 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:50.488+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:50.769 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:50.488+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.740364+0000 front 2024-09-10T19:21:28.740334+0000 (oldest deadline 2024-09-10T19:21:50.440168+0000) 2024-09-10T19:21:51.122 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:50 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:50.829+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:51.442 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:51 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:21:51.120+0000 7f08964d4640 -1 osd.4 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:17.905939+0000 front 2024-09-10T19:21:17.905834+0000 (oldest deadline 2024-09-10T19:21:43.805759+0000) 2024-09-10T19:21:51.443 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:51 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:51.298+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:51.768 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:51 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:51.440+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:51.768 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:51 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:51.440+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.740364+0000 front 2024-09-10T19:21:28.740334+0000 (oldest deadline 2024-09-10T19:21:50.440168+0000) 2024-09-10T19:21:52.083 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:51 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:51.805+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:52.474 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:52.271+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:52.475 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:21:52.081+0000 7f08964d4640 -1 osd.4 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:17.905939+0000 front 2024-09-10T19:21:17.905834+0000 (oldest deadline 2024-09-10T19:21:43.805759+0000) 2024-09-10T19:21:52.475 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4[37805]: 2024-09-10T19:21:52.081+0000 7f08964d4640 -1 osd.4 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.506897+0000 front 2024-09-10T19:21:28.506840+0000 (oldest deadline 2024-09-10T19:21:51.307332+0000) 2024-09-10T19:21:52.768 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:52.472+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:52.769 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:52.472+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.740364+0000 front 2024-09-10T19:21:28.740334+0000 (oldest deadline 2024-09-10T19:21:50.440168+0000) 2024-09-10T19:21:53.232 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:52 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:52.778+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:53.518 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:53 smithi194 podman[71931]: 2024-09-10 19:21:53.277074401 +0000 UTC m=+5.523756571 container died ed48d8435ed4ed50f9f89c344132faba36e1119c49aa3bdc72779d8b24e467f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T19:21:53.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:53 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:53.230+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:53.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:53 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:53.230+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:21:53.519 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:53 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:53.478+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:53.519 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:53 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:53.478+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.740364+0000 front 2024-09-10T19:21:28.740334+0000 (oldest deadline 2024-09-10T19:21:50.440168+0000) 2024-09-10T19:21:53.951 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:53 smithi194 podman[71931]: 2024-09-10 19:21:53.673304885 +0000 UTC m=+5.919987055 container cleanup ed48d8435ed4ed50f9f89c344132faba36e1119c49aa3bdc72779d8b24e467f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD) 2024-09-10T19:21:53.951 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:53 smithi194 bash[71931]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4 2024-09-10T19:21:53.951 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:53 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:53.779+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:53.952 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:53 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:53.780+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:21:54.067 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:53 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:53.571Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:21:54.067 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:53 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:53.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:21:54.067 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:21:53 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:21:53.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:21:54.258 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:53 smithi194 podman[71951]: 2024-09-10 19:21:53.949787998 +0000 UTC m=+0.663670979 container remove ed48d8435ed4ed50f9f89c344132faba36e1119c49aa3bdc72779d8b24e467f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True) 2024-09-10T19:21:54.509 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:54 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:54.256+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:54.509 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:54 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:54.256+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:21:54.509 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:54 smithi194 podman[72046]: 2024-09-10 19:21:54.183607227 +0000 UTC m=+0.015566979 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:21:54.509 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:54 smithi194 podman[72046]: 2024-09-10 19:21:54.292185557 +0000 UTC m=+0.124145304 container create 852a31aa4c5fff388ff2756a99a5bf5a3544bc6a2e52994b620e7f0f4467bd84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True) 2024-09-10T19:21:54.769 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:54 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:54.507+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:54.769 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:54 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:54.507+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.740364+0000 front 2024-09-10T19:21:28.740334+0000 (oldest deadline 2024-09-10T19:21:50.440168+0000) 2024-09-10T19:21:54.769 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:54 smithi194 podman[72046]: 2024-09-10 19:21:54.52790023 +0000 UTC m=+0.359859980 container init 852a31aa4c5fff388ff2756a99a5bf5a3544bc6a2e52994b620e7f0f4467bd84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-10T19:21:54.769 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:54 smithi194 podman[72046]: 2024-09-10 19:21:54.53297361 +0000 UTC m=+0.364933363 container start 852a31aa4c5fff388ff2756a99a5bf5a3544bc6a2e52994b620e7f0f4467bd84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-10T19:21:54.769 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:54 smithi194 podman[72046]: 2024-09-10 19:21:54.619604291 +0000 UTC m=+0.451564039 container attach 852a31aa4c5fff388ff2756a99a5bf5a3544bc6a2e52994b620e7f0f4467bd84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0) 2024-09-10T19:21:54.769 INFO:journalctl@ceph.osd.4.smithi194.stdout:Sep 10 19:21:54 smithi194 podman[72046]: 2024-09-10 19:21:54.708786591 +0000 UTC m=+0.540746341 container died 852a31aa4c5fff388ff2756a99a5bf5a3544bc6a2e52994b620e7f0f4467bd84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=) 2024-09-10T19:21:55.235 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:54 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:54.782+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:55.235 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:54 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:54.782+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:21:55.311 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.4.service' 2024-09-10T19:21:55.350 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:21:55.351 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-10T19:21:55.351 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-10T19:21:55.351 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.5 2024-09-10T19:21:55.494 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:55 smithi194 systemd[1]: Stopping Ceph osd.5 for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:21:55.494 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:55 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:55.233+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:55.494 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:55 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:55.233+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:21:55.768 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:55 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:55.491+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:55.769 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:55 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:55.491+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.740364+0000 front 2024-09-10T19:21:28.740334+0000 (oldest deadline 2024-09-10T19:21:50.440168+0000) 2024-09-10T19:21:55.769 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:55 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:55.677+0000 7f2e267e2640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:21:55.769 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:55 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:55.677+0000 7f2e267e2640 -1 osd.5 767 *** Got signal Terminated *** 2024-09-10T19:21:55.769 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:55 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:55.677+0000 7f2e267e2640 -1 osd.5 767 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:21:56.244 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:55 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:55.800+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:56.244 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:55 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:55.800+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:21:56.518 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:56 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:56.466+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:56.518 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:56 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:56.467+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.740364+0000 front 2024-09-10T19:21:28.740334+0000 (oldest deadline 2024-09-10T19:21:50.440168+0000) 2024-09-10T19:21:56.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:56 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:56.242+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:56.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:56 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:56.242+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:21:57.197 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:56 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:56.827+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:57.198 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:56 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:56.827+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:21:57.450 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:57 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:57.195+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:57.450 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:57 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:57.195+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:21:57.768 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:57 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:57.448+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:57.769 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:57 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:57.448+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.740364+0000 front 2024-09-10T19:21:28.740334+0000 (oldest deadline 2024-09-10T19:21:50.440168+0000) 2024-09-10T19:21:58.170 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:57 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:57.840+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:58.170 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:57 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:57.840+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:21:58.437 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:58.168+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:58.437 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:58.168+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:21:58.768 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:58.435+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:58.769 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:58.435+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.740364+0000 front 2024-09-10T19:21:28.740334+0000 (oldest deadline 2024-09-10T19:21:50.440168+0000) 2024-09-10T19:21:59.164 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:58.854+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:21:59.164 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:58.854+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:21:59.164 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:58 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:58.854+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:21:59.518 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:59 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:59.403+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:21:59.519 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:21:59 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:21:59.403+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.740364+0000 front 2024-09-10T19:21:28.740334+0000 (oldest deadline 2024-09-10T19:21:50.440168+0000) 2024-09-10T19:21:59.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:59 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:59.161+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:21:59.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:59 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:59.161+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:21:59.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:21:59 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:21:59.162+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:34.877724+0000 front 2024-09-10T19:21:34.877782+0000 (oldest deadline 2024-09-10T19:21:58.377589+0000) 2024-09-10T19:22:00.187 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:59 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:59.839+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:00.187 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:59 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:59.839+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:00.187 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:21:59 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:21:59.839+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:00.451 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:00.185+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:22:00.452 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:00.185+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:22:00.452 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:00.185+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:34.877724+0000 front 2024-09-10T19:21:34.877782+0000 (oldest deadline 2024-09-10T19:21:58.377589+0000) 2024-09-10T19:22:00.567 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:00 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:00.151Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:22:00.567 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:00 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:00.151Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:22:00.567 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:00 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:00.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:22:00.768 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:22:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:22:00.449+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:21.239029+0000 front 2024-09-10T19:21:21.239155+0000 (oldest deadline 2024-09-10T19:21:42.338911+0000) 2024-09-10T19:22:00.769 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:22:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5[44187]: 2024-09-10T19:22:00.449+0000 7f2e22dfa640 -1 osd.5 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.740364+0000 front 2024-09-10T19:21:28.740334+0000 (oldest deadline 2024-09-10T19:21:50.440168+0000) 2024-09-10T19:22:01.085 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:00.812+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:01.085 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:00.812+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:01.085 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:00 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:00.812+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:01.395 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:22:01 smithi194 podman[72175]: 2024-09-10 19:22:01.084392771 +0000 UTC m=+5.512402218 container died 8c5a964e60481a49e8545a3a5bbb43880f12d5f7b5016b4a77281b461fdded7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True) 2024-09-10T19:22:01.396 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:01 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:01.208+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:22:01.396 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:01 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:01.208+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:22:01.396 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:01 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:01.208+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:34.877724+0000 front 2024-09-10T19:21:34.877782+0000 (oldest deadline 2024-09-10T19:21:58.377589+0000) 2024-09-10T19:22:01.711 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:22:01 smithi194 podman[72175]: 2024-09-10 19:22:01.484011613 +0000 UTC m=+5.912021062 container cleanup 8c5a964e60481a49e8545a3a5bbb43880f12d5f7b5016b4a77281b461fdded7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-10T19:22:01.711 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:22:01 smithi194 bash[72175]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5 2024-09-10T19:22:02.018 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:22:01 smithi194 podman[72195]: 2024-09-10 19:22:01.709908779 +0000 UTC m=+0.624222738 container remove 8c5a964e60481a49e8545a3a5bbb43880f12d5f7b5016b4a77281b461fdded7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-10T19:22:02.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:01 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:01.766+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:02.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:01 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:01.766+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:02.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:01 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:01.766+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:02.277 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:02.231+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:22:02.277 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:02.231+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:22:02.277 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:02.231+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:34.877724+0000 front 2024-09-10T19:21:34.877782+0000 (oldest deadline 2024-09-10T19:21:58.377589+0000) 2024-09-10T19:22:02.278 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:22:02 smithi194 podman[72294]: 2024-09-10 19:22:01.927195661 +0000 UTC m=+0.015945371 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:22:02.278 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:22:02 smithi194 podman[72294]: 2024-09-10 19:22:02.042314737 +0000 UTC m=+0.131064434 container create 7121d204c0eb5b6d98810d898a64003b11d7495b5f157c902ca3347b0f544ae6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-10T19:22:02.278 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:22:02 smithi194 podman[72294]: 2024-09-10 19:22:02.271669544 +0000 UTC m=+0.360419244 container init 7121d204c0eb5b6d98810d898a64003b11d7495b5f157c902ca3347b0f544ae6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-10T19:22:02.278 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:22:02 smithi194 podman[72294]: 2024-09-10 19:22:02.276414054 +0000 UTC m=+0.365163751 container start 7121d204c0eb5b6d98810d898a64003b11d7495b5f157c902ca3347b0f544ae6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T19:22:02.745 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:22:02 smithi194 podman[72294]: 2024-09-10 19:22:02.371701852 +0000 UTC m=+0.460451566 container attach 7121d204c0eb5b6d98810d898a64003b11d7495b5f157c902ca3347b0f544ae6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5-deactivate, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-10T19:22:02.745 INFO:journalctl@ceph.osd.5.smithi194.stdout:Sep 10 19:22:02 smithi194 podman[72294]: 2024-09-10 19:22:02.454247687 +0000 UTC m=+0.542997392 container died 7121d204c0eb5b6d98810d898a64003b11d7495b5f157c902ca3347b0f544ae6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:22:03.018 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:02.742+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:03.018 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:02.742+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:03.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:02 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:02.742+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:03.057 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.5.service' 2024-09-10T19:22:03.095 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:22:03.095 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-10T19:22:03.095 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-10T19:22:03.096 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.6 2024-09-10T19:22:03.447 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:03 smithi194 systemd[1]: Stopping Ceph osd.6 for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:22:03.447 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:03.199+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:22:03.447 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:03.199+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:22:03.447 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:03.199+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:34.877724+0000 front 2024-09-10T19:21:34.877782+0000 (oldest deadline 2024-09-10T19:21:58.377589+0000) 2024-09-10T19:22:03.768 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:03.445+0000 7f3c33504640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:22:03.768 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:03.445+0000 7f3c33504640 -1 osd.6 767 *** Got signal Terminated *** 2024-09-10T19:22:03.769 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:03.445+0000 7f3c33504640 -1 osd.6 767 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:22:04.067 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:03 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:03.571Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:22:04.067 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:03 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:03.571Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:22:04.067 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:03 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:03.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:22:04.175 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:03.775+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:04.175 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:03.775+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:04.175 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:03 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:03.775+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:04.518 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:04.173+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:22:04.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:04.173+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:22:04.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:04.173+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:34.877724+0000 front 2024-09-10T19:21:34.877782+0000 (oldest deadline 2024-09-10T19:21:58.377589+0000) 2024-09-10T19:22:05.146 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:04.787+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:05.146 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:04.787+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:05.146 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:04 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:04.787+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:05.518 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:05 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:05.144+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:22:05.518 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:05 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:05.144+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:22:05.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:05 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:05.144+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:34.877724+0000 front 2024-09-10T19:21:34.877782+0000 (oldest deadline 2024-09-10T19:21:58.377589+0000) 2024-09-10T19:22:06.155 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:05 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:05.820+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:06.155 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:05 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:05.820+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:06.155 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:05 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:05.820+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:06.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:06 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:06.154+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:22:06.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:06 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:06.154+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:22:06.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:06 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:06.154+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:34.877724+0000 front 2024-09-10T19:21:34.877782+0000 (oldest deadline 2024-09-10T19:21:58.377589+0000) 2024-09-10T19:22:06.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:06 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:06.154+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6815 osd.3 since back 2024-09-10T19:21:43.579144+0000 front 2024-09-10T19:21:43.579036+0000 (oldest deadline 2024-09-10T19:22:05.879096+0000) 2024-09-10T19:22:07.175 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:06 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:06.810+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:07.176 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:06 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:06.810+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:07.176 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:06 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:06.810+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:07.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:07 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:07.174+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:22:07.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:07 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:07.174+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:22:07.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:07 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:07.174+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:34.877724+0000 front 2024-09-10T19:21:34.877782+0000 (oldest deadline 2024-09-10T19:21:58.377589+0000) 2024-09-10T19:22:07.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:07 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:07.174+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6815 osd.3 since back 2024-09-10T19:21:43.579144+0000 front 2024-09-10T19:21:43.579036+0000 (oldest deadline 2024-09-10T19:22:05.879096+0000) 2024-09-10T19:22:08.131 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:07 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:07.851+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:08.131 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:07 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:07.851+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:08.131 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:07 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:07.851+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:08.518 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:08.129+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:20.275646+0000 front 2024-09-10T19:21:20.275850+0000 (oldest deadline 2024-09-10T19:21:44.975812+0000) 2024-09-10T19:22:08.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:08.129+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:26.677115+0000 front 2024-09-10T19:21:26.677022+0000 (oldest deadline 2024-09-10T19:21:52.576836+0000) 2024-09-10T19:22:08.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:08.129+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:34.877724+0000 front 2024-09-10T19:21:34.877782+0000 (oldest deadline 2024-09-10T19:21:58.377589+0000) 2024-09-10T19:22:08.519 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6[50539]: 2024-09-10T19:22:08.129+0000 7f3c30b1e640 -1 osd.6 767 heartbeat_check: no reply from 172.21.15.119:6815 osd.3 since back 2024-09-10T19:21:43.579144+0000 front 2024-09-10T19:21:43.579036+0000 (oldest deadline 2024-09-10T19:22:05.879096+0000) 2024-09-10T19:22:09.088 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:08 smithi194 podman[72423]: 2024-09-10 19:22:08.836748893 +0000 UTC m=+5.519299315 container died dfcb8ee3a22307dbec60091dc3979df667d93ff94a8c44b2342395f7d4d9076b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-09-10T19:22:09.088 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:08.832+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:09.088 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:08.832+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:09.088 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:08 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:08.832+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:09.495 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:09 smithi194 podman[72423]: 2024-09-10 19:22:09.193560628 +0000 UTC m=+5.876111042 container cleanup dfcb8ee3a22307dbec60091dc3979df667d93ff94a8c44b2342395f7d4d9076b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:22:09.495 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:09 smithi194 bash[72423]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6 2024-09-10T19:22:09.768 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:09 smithi194 podman[72443]: 2024-09-10 19:22:09.494673366 +0000 UTC m=+0.660349247 container remove dfcb8ee3a22307dbec60091dc3979df667d93ff94a8c44b2342395f7d4d9076b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD) 2024-09-10T19:22:10.115 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:09 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:09.857+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:10.115 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:09 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:09.857+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:10.115 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:09 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:09.857+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:10.115 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:09 smithi194 podman[72536]: 2024-09-10 19:22:09.704056397 +0000 UTC m=+0.016091077 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:22:10.115 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:09 smithi194 podman[72536]: 2024-09-10 19:22:09.812357921 +0000 UTC m=+0.124392594 container create fa3970641fab41d00ab3b0e7d3867503d3e95bb792958585ed31f2d944846de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True) 2024-09-10T19:22:10.116 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:10 smithi194 podman[72536]: 2024-09-10 19:22:10.014688257 +0000 UTC m=+0.326722930 container init fa3970641fab41d00ab3b0e7d3867503d3e95bb792958585ed31f2d944846de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux ) 2024-09-10T19:22:10.116 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:10 smithi194 podman[72536]: 2024-09-10 19:22:10.019821903 +0000 UTC m=+0.331856576 container start fa3970641fab41d00ab3b0e7d3867503d3e95bb792958585ed31f2d944846de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-09-10T19:22:10.116 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:10 smithi194 podman[72536]: 2024-09-10 19:22:10.114334347 +0000 UTC m=+0.426369027 container attach fa3970641fab41d00ab3b0e7d3867503d3e95bb792958585ed31f2d944846de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-10T19:22:10.476 INFO:journalctl@ceph.osd.6.smithi194.stdout:Sep 10 19:22:10 smithi194 podman[72536]: 2024-09-10 19:22:10.193301871 +0000 UTC m=+0.505336555 container died fa3970641fab41d00ab3b0e7d3867503d3e95bb792958585ed31f2d944846de3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-10T19:22:10.567 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:10 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:10.151Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:22:10.567 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:10 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:10.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:22:10.567 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:10 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:10.152Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:22:10.756 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.6.service' 2024-09-10T19:22:10.794 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:22:10.794 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-10T19:22:10.794 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-10T19:22:10.794 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.7 2024-09-10T19:22:11.018 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:10.864+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:11.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:10.864+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:11.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:10.864+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:11.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:10 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:10.864+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6815 osd.3 since back 2024-09-10T19:21:44.034382+0000 front 2024-09-10T19:21:44.034416+0000 (oldest deadline 2024-09-10T19:22:09.934404+0000) 2024-09-10T19:22:11.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:10 smithi194 systemd[1]: Stopping Ceph osd.7 for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:22:11.519 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:11 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:11.113+0000 7f77799aa640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:22:11.519 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:11 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:11.113+0000 7f77799aa640 -1 osd.7 767 *** Got signal Terminated *** 2024-09-10T19:22:11.519 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:11 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:11.113+0000 7f77799aa640 -1 osd.7 767 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:22:12.269 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:11 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:11.815+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:12.269 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:11 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:11.815+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:12.269 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:11 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:11.815+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:12.269 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:11 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:11.815+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6815 osd.3 since back 2024-09-10T19:21:44.034382+0000 front 2024-09-10T19:21:44.034416+0000 (oldest deadline 2024-09-10T19:22:09.934404+0000) 2024-09-10T19:22:13.018 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:12.770+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:13.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:12.770+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:13.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:12.770+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:13.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:12 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:12.770+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6815 osd.3 since back 2024-09-10T19:21:44.034382+0000 front 2024-09-10T19:21:44.034416+0000 (oldest deadline 2024-09-10T19:22:09.934404+0000) 2024-09-10T19:22:14.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:13 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:13.763+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:14.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:13 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:13.763+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:14.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:13 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:13.763+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:14.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:13 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:13.763+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6815 osd.3 since back 2024-09-10T19:21:44.034382+0000 front 2024-09-10T19:21:44.034416+0000 (oldest deadline 2024-09-10T19:22:09.934404+0000) 2024-09-10T19:22:14.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:13 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:13.763+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.194:6802 osd.4 since back 2024-09-10T19:21:52.235295+0000 front 2024-09-10T19:21:52.235334+0000 (oldest deadline 2024-09-10T19:22:13.335415+0000) 2024-09-10T19:22:14.067 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:13 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:13.572Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=3 err="ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:22:14.067 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:13 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:13.572Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:22:14.067 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:13 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:13.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:22:15.018 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:14 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:14.750+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:15.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:14 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:14.750+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:15.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:14 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:14.750+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:15.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:14 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:14.750+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6815 osd.3 since back 2024-09-10T19:21:44.034382+0000 front 2024-09-10T19:21:44.034416+0000 (oldest deadline 2024-09-10T19:22:09.934404+0000) 2024-09-10T19:22:15.019 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:14 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:14.750+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.194:6802 osd.4 since back 2024-09-10T19:21:52.235295+0000 front 2024-09-10T19:21:52.235334+0000 (oldest deadline 2024-09-10T19:22:13.335415+0000) 2024-09-10T19:22:16.269 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:15.800+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6803 osd.0 since back 2024-09-10T19:21:19.229682+0000 front 2024-09-10T19:21:19.229640+0000 (oldest deadline 2024-09-10T19:21:43.929689+0000) 2024-09-10T19:22:16.269 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:15.800+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6807 osd.1 since back 2024-09-10T19:21:28.332341+0000 front 2024-09-10T19:21:28.332327+0000 (oldest deadline 2024-09-10T19:21:53.032227+0000) 2024-09-10T19:22:16.269 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:15.800+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6811 osd.2 since back 2024-09-10T19:21:35.833577+0000 front 2024-09-10T19:21:35.833440+0000 (oldest deadline 2024-09-10T19:21:58.733484+0000) 2024-09-10T19:22:16.269 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:15.800+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.119:6815 osd.3 since back 2024-09-10T19:21:44.034382+0000 front 2024-09-10T19:21:44.034416+0000 (oldest deadline 2024-09-10T19:22:09.934404+0000) 2024-09-10T19:22:16.269 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:15 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7[56794]: 2024-09-10T19:22:15.800+0000 7f77747bf640 -1 osd.7 767 heartbeat_check: no reply from 172.21.15.194:6802 osd.4 since back 2024-09-10T19:21:52.235295+0000 front 2024-09-10T19:21:52.235334+0000 (oldest deadline 2024-09-10T19:22:13.335415+0000) 2024-09-10T19:22:16.970 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:16 smithi194 podman[72665]: 2024-09-10 19:22:16.664819562 +0000 UTC m=+5.653371844 container died 11277776ca4f9ecae2df11c90363345ae9946a29c19d9776a832f41eb2c78bd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-10T19:22:17.268 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:17 smithi194 podman[72665]: 2024-09-10 19:22:17.053132571 +0000 UTC m=+6.041684854 container cleanup 11277776ca4f9ecae2df11c90363345ae9946a29c19d9776a832f41eb2c78bd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:22:17.269 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:17 smithi194 bash[72665]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7 2024-09-10T19:22:17.617 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:17 smithi194 podman[72700]: 2024-09-10 19:22:17.287440518 +0000 UTC m=+0.620219560 container remove 11277776ca4f9ecae2df11c90363345ae9946a29c19d9776a832f41eb2c78bd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-10T19:22:17.899 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:17 smithi194 podman[72790]: 2024-09-10 19:22:17.519367858 +0000 UTC m=+0.015816272 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:22:17.900 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:17 smithi194 podman[72790]: 2024-09-10 19:22:17.621561281 +0000 UTC m=+0.118009751 container create 531f63cea279d2a54e90b0f7f8414bebdc6abd88f75ca7fa483b595e3283ad7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True) 2024-09-10T19:22:17.900 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:17 smithi194 podman[72790]: 2024-09-10 19:22:17.815666155 +0000 UTC m=+0.312114569 container init 531f63cea279d2a54e90b0f7f8414bebdc6abd88f75ca7fa483b595e3283ad7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-10T19:22:17.900 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:17 smithi194 podman[72790]: 2024-09-10 19:22:17.82068555 +0000 UTC m=+0.317133972 container start 531f63cea279d2a54e90b0f7f8414bebdc6abd88f75ca7fa483b595e3283ad7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:22:17.900 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:17 smithi194 podman[72790]: 2024-09-10 19:22:17.898569938 +0000 UTC m=+0.395018371 container attach 531f63cea279d2a54e90b0f7f8414bebdc6abd88f75ca7fa483b595e3283ad7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2) 2024-09-10T19:22:18.269 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:17 smithi194 podman[72790]: 2024-09-10 19:22:17.989142698 +0000 UTC m=+0.485591113 container died 531f63cea279d2a54e90b0f7f8414bebdc6abd88f75ca7fa483b595e3283ad7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7-deactivate, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:22:18.517 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.7.service' 2024-09-10T19:22:18.541 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:18 smithi194 podman[72834]: 2024-09-10 19:22:18.49301572 +0000 UTC m=+0.494174683 container remove 531f63cea279d2a54e90b0f7f8414bebdc6abd88f75ca7fa483b595e3283ad7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-osd-7-deactivate, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:22:18.541 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:18 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.7.service: Deactivated successfully. 2024-09-10T19:22:18.541 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:18 smithi194 systemd[1]: Stopped Ceph osd.7 for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:22:18.541 INFO:journalctl@ceph.osd.7.smithi194.stdout:Sep 10 19:22:18 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@osd.7.service: Consumed 35.007s CPU time. 2024-09-10T19:22:18.558 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:22:18.558 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-10T19:22:18.558 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-10T19:22:18.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@rgw.foo.a 2024-09-10T19:22:18.912 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:22:18 smithi119 systemd[1]: Stopping Ceph rgw.foo.a for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:22:19.316 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:22:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-rgw-foo-a[68766]: 2024-09-10T19:22:18.909+0000 7f22577fe640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:22:19.317 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:22:18 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-rgw-foo-a[68766]: 2024-09-10T19:22:18.909+0000 7f22683f27c0 -1 shutting down 2024-09-10T19:22:20.567 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:20 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:20.152Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:22:20.567 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:20 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:20.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:22:20.567 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:20 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:20.153Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:22:24.067 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:23 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:23.572Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=4 err="ceph-dashboard/webhook[1]: notify retry canceled after 7 attempts: Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused; ceph-dashboard/webhook[0]: notify retry canceled after 7 attempts: Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:22:24.067 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:23 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:23.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:22:24.067 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:23 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:23.573Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:22:29.317 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:22:28 smithi119 bash[151007]: time="2024-09-10T19:22:28Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-10T19:22:29.317 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:22:28 smithi119 podman[151007]: 2024-09-10 19:22:28.933284076 +0000 UTC m=+10.179484682 container died a3ed74e0e69880090c038472ffc6319548a1b31baa7921e627b36553ae9c9f91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-rgw-foo-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:22:29.650 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:22:29 smithi119 podman[151007]: 2024-09-10 19:22:29.46551649 +0000 UTC m=+10.711717094 container cleanup a3ed74e0e69880090c038472ffc6319548a1b31baa7921e627b36553ae9c9f91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-rgw-foo-a, io.buildah.version=1.37.2, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-10T19:22:29.651 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:22:29 smithi119 bash[151007]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-rgw-foo-a 2024-09-10T19:22:29.894 DEBUG:teuthology.orchestra.run.smithi119:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@rgw.foo.a.service' 2024-09-10T19:22:29.916 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:22:29 smithi119 podman[151033]: 2024-09-10 19:22:29.649752987 +0000 UTC m=+0.714119326 container remove a3ed74e0e69880090c038472ffc6319548a1b31baa7921e627b36553ae9c9f91 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-rgw-foo-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-09-10T19:22:29.916 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:22:29 smithi119 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-09-10T19:22:29.916 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:22:29 smithi119 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@rgw.foo.a.service: Failed with result 'exit-code'. 2024-09-10T19:22:29.916 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:22:29 smithi119 systemd[1]: Stopped Ceph rgw.foo.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:22:29.916 INFO:journalctl@ceph.rgw.foo.a.smithi119.stdout:Sep 10 19:22:29 smithi119 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@rgw.foo.a.service: Consumed 3.758s CPU time. 2024-09-10T19:22:29.930 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:22:29.930 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-10T19:22:29.930 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-10T19:22:29.930 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@prometheus.a 2024-09-10T19:22:30.231 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:29 smithi194 systemd[1]: Stopping Ceph prometheus.a for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:22:30.317 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:30 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:30.153Z caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="ceph-dashboard/webhook[0]: notify retry canceled after 8 attempts: Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused; ceph-dashboard/webhook[1]: notify retry canceled after 8 attempts: Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:22:30.317 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:30 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:30.154Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:22:30.317 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:30 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:30.154Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:22:30.497 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:22:30.230Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-10T19:22:30.497 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:22:30.230Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-10T19:22:30.497 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:22:30.230Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-10T19:22:30.497 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:22:30.230Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-10T19:22:30.497 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:22:30.230Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-10T19:22:30.497 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:22:30.230Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-10T19:22:30.497 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:22:30.230Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-10T19:22:30.497 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:22:30.230Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-10T19:22:30.497 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:22:30.230Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-10T19:22:30.498 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:22:30.231Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-10T19:22:30.498 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:22:30.231Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-10T19:22:30.498 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a[67310]: ts=2024-09-10T19:22:30.231Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-10T19:22:30.498 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 podman[72920]: 2024-09-10 19:22:30.241144767 +0000 UTC m=+0.121120059 container died ea606254366cec508b61d18b29d322816fc1e209da1e6cc57c7b8e4ed18af53a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:22:30.769 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 podman[72920]: 2024-09-10 19:22:30.594621412 +0000 UTC m=+0.474596691 container cleanup ea606254366cec508b61d18b29d322816fc1e209da1e6cc57c7b8e4ed18af53a (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:22:30.769 INFO:journalctl@ceph.prometheus.a.smithi194.stdout:Sep 10 19:22:30 smithi194 bash[72920]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-prometheus-a 2024-09-10T19:22:31.026 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@prometheus.a.service' 2024-09-10T19:22:31.060 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:22:31.060 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-10T19:22:31.061 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm rm-cluster --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-10T19:22:31.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:31 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:31.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi119.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.119:8443: connect: connection refused" 2024-09-10T19:22:31.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:31 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:31.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi194.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.194:8443: connect: connection refused" 2024-09-10T19:22:32.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:32 smithi119 systemd[1]: Stopping Ceph alertmanager.a for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:22:33.223 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:32 smithi119 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a[78175]: ts=2024-09-10T19:22:32.939Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-10T19:22:33.223 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:32 smithi119 podman[151462]: 2024-09-10 19:22:32.980445825 +0000 UTC m=+0.134777740 container died 1b10ef1281739cdf6e840b404c3b9b655877e92e0e3816b79b9e01e11e39f35f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:22:33.522 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:33 smithi119 podman[151462]: 2024-09-10 19:22:33.320777312 +0000 UTC m=+0.475109231 container cleanup 1b10ef1281739cdf6e840b404c3b9b655877e92e0e3816b79b9e01e11e39f35f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:22:33.522 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:33 smithi119 bash[151462]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a 2024-09-10T19:22:33.816 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:33 smithi119 podman[151479]: 2024-09-10 19:22:33.521556134 +0000 UTC m=+0.538201422 container remove 1b10ef1281739cdf6e840b404c3b9b655877e92e0e3816b79b9e01e11e39f35f (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:22:33.817 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:33 smithi119 podman[151479]: 2024-09-10 19:22:33.60551118 +0000 UTC m=+0.622156469 volume remove 71aecd5e36d7b9b3839a7b39bb8e3ba462f62410db9199a8f8e59fcd33c34a47 2024-09-10T19:22:34.316 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:22:34 smithi119 systemd[1]: Stopping Ceph node-exporter.a for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:22:34.317 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:33 smithi119 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-10T19:22:34.317 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:33 smithi119 systemd[1]: Stopped Ceph alertmanager.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:22:34.317 INFO:journalctl@ceph.alertmanager.a.smithi119.stdout:Sep 10 19:22:33 smithi119 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@alertmanager.a.service: Consumed 1.811s CPU time. 2024-09-10T19:22:34.576 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:22:34 smithi119 podman[151591]: 2024-09-10 19:22:34.320847526 +0000 UTC m=+0.119857226 container died e8af2e4da1f4568a6277b6e396f0cf51f34155200d269b8cb57e62f93d0ea959 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T19:22:34.960 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:22:34 smithi119 podman[151591]: 2024-09-10 19:22:34.666394562 +0000 UTC m=+0.465404323 container cleanup e8af2e4da1f4568a6277b6e396f0cf51f34155200d269b8cb57e62f93d0ea959 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T19:22:34.960 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:22:34 smithi119 bash[151591]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a 2024-09-10T19:22:35.260 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:22:34 smithi119 podman[151608]: 2024-09-10 19:22:34.959495311 +0000 UTC m=+0.642593465 container remove e8af2e4da1f4568a6277b6e396f0cf51f34155200d269b8cb57e62f93d0ea959 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T19:22:35.260 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:22:34 smithi119 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-10T19:22:35.260 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:22:35 smithi119 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-10T19:22:35.260 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:22:35 smithi119 systemd[1]: Stopped Ceph node-exporter.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:22:35.261 INFO:journalctl@ceph.node-exporter.a.smithi119.stdout:Sep 10 19:22:35 smithi119 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@node-exporter.a.service: Consumed 3.517s CPU time. 2024-09-10T19:22:35.746 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm rm-cluster --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-10T19:22:37.519 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:37 smithi194 systemd[1]: Stopping Ceph iscsi.iscsi.a for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:22:37.519 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:37 smithi194 ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a[61949]: debug Shutdown received 2024-09-10T19:22:47.660 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:47 smithi194 bash[73301]: time="2024-09-10T19:22:47Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-10T19:22:47.660 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:47 smithi194 podman[73301]: 2024-09-10 19:22:47.386597087 +0000 UTC m=+10.134304162 container died 5fe7fd33d0dba93c2142639b5898165895b89af0b1aa38dd8338d92422d6ee15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-10T19:22:47.915 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:47 smithi194 podman[73301]: 2024-09-10 19:22:47.732747781 +0000 UTC m=+10.480454849 container cleanup 5fe7fd33d0dba93c2142639b5898165895b89af0b1aa38dd8338d92422d6ee15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-10T19:22:47.915 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:47 smithi194 bash[73301]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a 2024-09-10T19:22:48.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:48 smithi194 podman[73325]: 2024-09-10 19:22:48.017199387 +0000 UTC m=+0.630388825 container remove 5fe7fd33d0dba93c2142639b5898165895b89af0b1aa38dd8338d92422d6ee15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-09-10T19:22:48.269 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:48 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-10T19:22:48.595 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:48 smithi194 podman[73382]: 2024-09-10 19:22:48.273595702 +0000 UTC m=+0.111979254 container died cd8cc8a1052ef878e0e2849458883a58267fefd097e9850193ce33f53516f541 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-09-10T19:22:48.595 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:48 smithi194 podman[73382]: 2024-09-10 19:22:48.594227728 +0000 UTC m=+0.432611276 container cleanup cd8cc8a1052ef878e0e2849458883a58267fefd097e9850193ce33f53516f541 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-10T19:22:48.852 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:48 smithi194 bash[73382]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu 2024-09-10T19:22:48.852 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:48 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-10T19:22:48.852 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:48 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Unit process 61790 (conmon) remains running after unit stopped. 2024-09-10T19:22:48.852 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:48 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Unit process 73399 (podman) remains running after unit stopped. 2024-09-10T19:22:48.852 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:48 smithi194 systemd[1]: Stopped Ceph iscsi.iscsi.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:22:48.852 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:48 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Consumed 3.298s CPU time. 2024-09-10T19:22:48.852 INFO:journalctl@ceph.iscsi.iscsi.a.smithi194.stdout:Sep 10 19:22:48 smithi194 podman[73399]: 2024-09-10 19:22:48.817001989 +0000 UTC m=+0.546265777 container remove cd8cc8a1052ef878e0e2849458883a58267fefd097e9850193ce33f53516f541 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-10T19:22:49.268 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:22:49 smithi194 systemd[1]: Stopping Ceph grafana.a for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:22:49.530 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:22:49 smithi194 podman[73525]: 2024-09-10 19:22:49.274411835 +0000 UTC m=+0.120196539 container died f222ca35a17b7d11dd64c78504b7d0f7961ec581985bb3fd34ce5372dbae6d41 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, build-date=2022-04-29T11:48:11.055897, name=ubi8, io.openshift.tags=base rhel8, io.buildah.version=1.32.0, vendor=Red Hat, Inc., architecture=x86_64, vcs-type=git, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, release=239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , version=8.5) 2024-09-10T19:22:49.806 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:22:49 smithi194 podman[73525]: 2024-09-10 19:22:49.603796621 +0000 UTC m=+0.449581316 container cleanup f222ca35a17b7d11dd64c78504b7d0f7961ec581985bb3fd34ce5372dbae6d41 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, architecture=x86_64, name=ubi8, version=8.5, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, vcs-type=git, vendor=Red Hat, Inc., description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, release=239.1651231664, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container) 2024-09-10T19:22:49.806 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:22:49 smithi194 bash[73525]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a 2024-09-10T19:22:50.206 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:22:49 smithi194 podman[73541]: 2024-09-10 19:22:49.805043883 +0000 UTC m=+0.529176250 container remove f222ca35a17b7d11dd64c78504b7d0f7961ec581985bb3fd34ce5372dbae6d41 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-grafana-a, io.buildah.version=1.32.0, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, vcs-type=git, com.redhat.component=ubi8-container, io.openshift.expose-services=, description=Ceph Grafana Container, name=ubi8, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, release=239.1651231664) 2024-09-10T19:22:50.206 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:22:49 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-10T19:22:50.206 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:22:50 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-10T19:22:50.206 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:22:50 smithi194 systemd[1]: Stopped Ceph grafana.a for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:22:50.206 INFO:journalctl@ceph.grafana.a.smithi194.stdout:Sep 10 19:22:50 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@grafana.a.service: Consumed 11.133s CPU time. 2024-09-10T19:22:50.519 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:22:50 smithi194 systemd[1]: Stopping Ceph node-exporter.b for 3afe9804-6fa6-11ef-bcea-c7b262605968... 2024-09-10T19:22:50.519 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:22:50 smithi194 podman[73651]: 2024-09-10 19:22:50.445786614 +0000 UTC m=+0.105930973 container died 87647be4cef28eaca4a224f5fa911d43dbf3e90e7a4e03b8c3a2298f9ba1db9f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T19:22:51.018 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:22:50 smithi194 podman[73651]: 2024-09-10 19:22:50.800008563 +0000 UTC m=+0.460152921 container cleanup 87647be4cef28eaca4a224f5fa911d43dbf3e90e7a4e03b8c3a2298f9ba1db9f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T19:22:51.019 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:22:50 smithi194 bash[73651]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b 2024-09-10T19:22:51.378 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:22:51 smithi194 podman[73668]: 2024-09-10 19:22:51.059552731 +0000 UTC m=+0.614176698 container remove 87647be4cef28eaca4a224f5fa911d43dbf3e90e7a4e03b8c3a2298f9ba1db9f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-3afe9804-6fa6-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T19:22:51.378 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:22:51 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-10T19:22:51.378 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:22:51 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-10T19:22:51.378 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:22:51 smithi194 systemd[1]: Stopped Ceph node-exporter.b for 3afe9804-6fa6-11ef-bcea-c7b262605968. 2024-09-10T19:22:51.378 INFO:journalctl@ceph.node-exporter.b.smithi194.stdout:Sep 10 19:22:51 smithi194 systemd[1]: ceph-3afe9804-6fa6-11ef-bcea-c7b262605968@node-exporter.b.service: Consumed 3.595s CPU time. 2024-09-10T19:22:51.874 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:22:51.908 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:22:51.935 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-10T19:22:51.937 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900276/remote/smithi119/crash 2024-09-10T19:22:51.938 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/crash -- . 2024-09-10T19:22:51.985 INFO:teuthology.orchestra.run.smithi119.stderr:tar: /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-10T19:22:51.985 INFO:teuthology.orchestra.run.smithi119.stderr:tar: Error is not recoverable: exiting now 2024-09-10T19:22:51.987 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900276/remote/smithi194/crash 2024-09-10T19:22:51.988 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/crash -- . 2024-09-10T19:22:52.025 INFO:teuthology.orchestra.run.smithi194.stderr:tar: /var/lib/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-10T19:22:52.025 INFO:teuthology.orchestra.run.smithi194.stderr:tar: Error is not recoverable: exiting now 2024-09-10T19:22:52.026 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-10T19:22:52.027 DEBUG:teuthology.orchestra.run.smithi119:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-09-10T19:22:52.055 INFO:teuthology.orchestra.run.smithi119.stderr:grep: /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph.log: No such file or directory 2024-09-10T19:22:52.058 INFO:tasks.cephadm:Compressing logs... 2024-09-10T19:22:52.058 DEBUG:teuthology.orchestra.run.smithi119:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T19:22:52.100 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T19:22:52.124 INFO:teuthology.orchestra.run.smithi119.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.0.log 2024-09-10T19:22:52.124 INFO:teuthology.orchestra.run.smithi119.stderr:: No such file or directory 2024-09-10T19:22:52.124 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.1.log 2024-09-10T19:22:52.124 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-10T19:22:52.124 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mon.c.log 2024-09-10T19:22:52.127 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mon.a.log 2024-09-10T19:22:52.127 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.2.log 2024-09-10T19:22:52.127 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mgr.y.log 2024-09-10T19:22:52.128 INFO:teuthology.orchestra.run.smithi194.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/tcmu-runner.log 2024-09-10T19:22:52.128 INFO:teuthology.orchestra.run.smithi194.stderr:: No such file or directory 2024-09-10T19:22:52.128 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.7.log 2024-09-10T19:22:52.128 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-10T19:22:52.128 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/tcmu-runner.log: 64.1% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/tcmu-runner.log.gz 2024-09-10T19:22:52.128 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.4.log 2024-09-10T19:22:52.129 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.6.log 2024-09-10T19:22:52.129 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.5.log 2024-09-10T19:22:52.129 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mgr.x.log 2024-09-10T19:22:52.129 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mon.b.log 2024-09-10T19:22:52.129 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-10T19:22:52.129 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 92.5% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-10T19:22:52.130 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log 2024-09-10T19:22:52.130 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mgr.y.log: /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.3.log 2024-09-10T19:22:52.136 INFO:teuthology.orchestra.run.smithi194.stderr: 92.1% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mgr.x.log.gz 2024-09-10T19:22:52.138 INFO:teuthology.orchestra.run.smithi194.stderr: 90.9% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-10T19:22:52.139 INFO:teuthology.orchestra.run.smithi119.stderr: 91.0% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-10T19:22:52.143 INFO:teuthology.orchestra.run.smithi119.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-10T19:22:52.163 INFO:teuthology.orchestra.run.smithi119.stderr:/var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.3.log: /var/log/ceph/cephadm.log: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-10T19:22:52.174 INFO:teuthology.orchestra.run.smithi119.stderr: 93.9% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-09-10T19:22:52.536 INFO:teuthology.orchestra.run.smithi119.stderr: 91.0% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mgr.y.log.gz 2024-09-10T19:22:53.150 INFO:teuthology.orchestra.run.smithi119.stderr: 92.8% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mon.c.log.gz 2024-09-10T19:22:53.382 INFO:teuthology.orchestra.run.smithi194.stderr: 92.8% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mon.b.log.gz 2024-09-10T19:22:54.139 INFO:teuthology.orchestra.run.smithi119.stderr: 91.9% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-mon.a.log.gz 2024-09-10T19:22:57.230 INFO:teuthology.orchestra.run.smithi194.stderr: 93.9% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.6.log.gz 2024-09-10T19:22:57.595 INFO:teuthology.orchestra.run.smithi194.stderr: 94.0% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.4.log.gz 2024-09-10T19:22:57.773 INFO:teuthology.orchestra.run.smithi194.stderr: 94.0% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.5.log.gz 2024-09-10T19:22:57.814 INFO:teuthology.orchestra.run.smithi119.stderr: 94.0% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.2.log.gz 2024-09-10T19:22:58.135 INFO:teuthology.orchestra.run.smithi119.stderr: 94.0% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.0.log.gz 2024-09-10T19:22:58.278 INFO:teuthology.orchestra.run.smithi119.stderr: 94.0% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.1.log.gz 2024-09-10T19:22:58.296 INFO:teuthology.orchestra.run.smithi194.stderr: 94.1% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.7.log.gz 2024-09-10T19:22:58.298 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-09-10T19:22:58.298 INFO:teuthology.orchestra.run.smithi194.stderr:real 0m6.183s 2024-09-10T19:22:58.298 INFO:teuthology.orchestra.run.smithi194.stderr:user 0m22.754s 2024-09-10T19:22:58.298 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m0.863s 2024-09-10T19:22:58.487 INFO:teuthology.orchestra.run.smithi119.stderr: 94.1% -- replaced with /var/log/ceph/3afe9804-6fa6-11ef-bcea-c7b262605968/ceph-osd.3.log.gz 2024-09-10T19:22:58.489 INFO:teuthology.orchestra.run.smithi119.stderr: 2024-09-10T19:22:58.489 INFO:teuthology.orchestra.run.smithi119.stderr:real 0m6.375s 2024-09-10T19:22:58.489 INFO:teuthology.orchestra.run.smithi119.stderr:user 0m26.696s 2024-09-10T19:22:58.489 INFO:teuthology.orchestra.run.smithi119.stderr:sys 0m0.982s 2024-09-10T19:22:58.489 INFO:tasks.cephadm:Archiving logs... 2024-09-10T19:22:58.489 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900276/remote/smithi119/log 2024-09-10T19:22:58.490 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T19:23:02.061 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900276/remote/smithi194/log 2024-09-10T19:23:02.062 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T19:23:05.212 INFO:tasks.cephadm:Removing cluster... 2024-09-10T19:23:05.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo cephadm rm-cluster --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 --force 2024-09-10T19:23:05.811 DEBUG:teuthology.orchestra.run.smithi194:> sudo cephadm rm-cluster --fsid 3afe9804-6fa6-11ef-bcea-c7b262605968 --force 2024-09-10T19:23:06.431 INFO:tasks.cephadm:Teardown complete 2024-09-10T19:23:06.431 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T19:23:06.441 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T19:23:06.441 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T19:23:06.444 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T19:23:06.511 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T19:23:06.519 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T19:23:06.520 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T19:23:06.524 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T19:23:06.535 INFO:teuthology.orchestra.run.smithi119.stderr:bash: line 1: ntpq: command not found 2024-09-10T19:23:06.538 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-09-10T19:23:06.539 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T19:23:06.539 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2024-09-10T19:23:06.539 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv01.front.sepia.ceph.com 3 6 377 54 -328us[ -316us] +/- 38ms 2024-09-10T19:23:06.539 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 53 +1439us[+1439us] +/- 56ms 2024-09-10T19:23:06.539 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 55 -1323us[-1311us] +/- 82ms 2024-09-10T19:23:06.539 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T19:23:06.542 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T19:23:06.543 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-09-10T19:23:06.543 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv01.front.sepia.ceph.com 3 6 377 55 -319us[ -316us] +/- 38ms 2024-09-10T19:23:06.543 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 56 +1481us[+1484us] +/- 56ms 2024-09-10T19:23:06.543 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 -1297us[-1297us] +/- 82ms 2024-09-10T19:23:06.543 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T19:23:06.543 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T19:23:06.552 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T19:23:06.552 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T19:23:06.560 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T19:23:06.658 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T19:23:06.729 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T19:23:06.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:06.758 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T19:23:06.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:06.785 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T19:23:06.794 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T19:23:06.802 INFO:teuthology.task.internal:Duration was 2030.680370 seconds 2024-09-10T19:23:06.803 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T19:23:06.811 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T19:23:06.811 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T19:23:06.813 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T19:23:06.849 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T19:23:06.864 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T19:23:07.104 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-10T19:23:07.104 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-09-10T19:23:07.104 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T19:23:07.164 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-09-10T19:23:07.164 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T19:23:07.201 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-10T19:23:07.201 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T19:23:07.207 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T19:23:07.266 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-10T19:23:07.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T19:23:07.274 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T19:23:07.689 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T19:23:07.699 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T19:23:07.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T19:23:07.723 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T19:23:07.749 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T19:23:07.758 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:23:07.766 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:23:07.786 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-09-10T19:23:07.812 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-09-10T19:23:07.835 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:23:07.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:07.854 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:23:07.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:07.890 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T19:23:07.900 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T19:23:07.901 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900276/remote/smithi119 2024-09-10T19:23:07.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T19:23:07.942 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900276/remote/smithi194 2024-09-10T19:23:07.942 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T19:23:07.978 INFO:teuthology.task.internal:Removing archive directory... 2024-09-10T19:23:07.978 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T19:23:07.981 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T19:23:08.027 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T19:23:08.037 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T19:23:08.037 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T19:23:08.046 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T19:23:08.046 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T19:23:08.049 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T19:23:08.061 INFO:teuthology.orchestra.run.smithi119.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 19:23 /home/ubuntu/cephtest 2024-09-10T19:23:08.081 INFO:teuthology.orchestra.run.smithi194.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 19:23 /home/ubuntu/cephtest 2024-09-10T19:23:08.082 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T19:23:08.092 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T19:23:08.112 INFO:teuthology.run:Summary data: description: orch:cephadm/with-work/{0-distro/centos_9.stream_runc fixed-2 mode/packaged mon_election/classic msgr/async-v2only start tasks/rados_api_tests} duration: 2030.6803703308105 flavor: default owner: scheduled_gabrioux@teuthology success: true 2024-09-10T19:23:08.112 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T19:23:08.185 INFO:teuthology.run:pass